Untitled  Document 


Files  are  in  Adobe  and  Microsoft  Word  Doc  format . 
Download  the  newest  version  from  Adobe . 


10th  ANNUAL  CMMI  TECHNOLOGY  CONFERENCE  AND  USER  GROUP 

"Investigation,  Measures,  and  Lessons  Learned  About  the  Relationship  Between  CMMI  Process 

Capability  and  Project  or  Program  Performance" 

Denver,  CO 

15  -  18  November  2010 

Agenda 

Tuesdayf  November  16f  2010 


Keynote  Speaker 

Mr.  Terence  Edwards,  Director,  System-of-Systems  Engineering  (SoSE),  ASA(ALT) 

CMMI®  Implementation  Strategies  for  Success  Panel 

•  Ms.  Nancy  Fleishcher,  Raytheon  Company 

•  Mr.  Rick  Bennett,  Director  of  Development,  MedPlus 

•  Mr.  Steve  Austin,  Lockheed  Martin  Mission 

CMMI®  VI. 3  Overview 

•  Mr.  Mike  Phillips,  Software  Engineering  Institute 

BENEFITS  OF  CMMI 

•  Mr.  Hal  Wilson,  Director,  Engineering,  Defense  Systems  Division 

Concurrent  Sessions 

TRACK  1-  CHASM  CREEK-  CMMI  AND  PROCESS  IMPROVEMENT 

•  1 1314  -  CMMI ®  in  the  Social  Media  (For  the  Social  Media-Challenged!),  Mr.  Bill  Smith,  Leading  Edge  Process  Consultants,  LLC 

•  1 13 15  -  What  Doesn  't  Kill  You  Makes  You  Stronger:  My  Process  Improvement  Lessons  Learned ,  Mr.  Bill  Smith,  Leading  Edge  Process 

Consultants,  LLC 

•  1 13 1 8  -  NASA ’s  Use  of  the  CMMI ®  for  Software  Engineering  Improvement ,  Dr.  John  Kelly,  NASA 

•  1 1297  -  Applying  Theory  of  Constraints  Tools  to  Focus  Lean  Development ,  Mr.  Tim  Oltman,  The  Boeing  Company 

TRACK  2  -  WINDSTAR  -  PRACTICAL  GUIDANCE 

•  10963  -  Configuration  Management ,  Mr.  A1  Florence,  The  MITRE  Corporation 

•  10970  -  Using  CMMI®  Effectively  in  Small  Business ,  NDIA  CMMI®  Working  Group 

•  1 1 1 58  -  Process  Alignment — From  Acquired  to  Acquirer ,  Ms.  Ilene  Pinsker,  Rockwell  Collins 

TRACK  3  -  MESA  VERDE  -  PERFORMANCE  RESULTS 

•  1 1307  -  Process  Performance  Models  for  Hardware  Engineers ,  Mr.  Tom  Fosness,  Northrop  Grumman  Corporation 

•  1 1 185  -  Navigating  the  Waters  of  SCAMPI  Reappraisal,  Mr.  Craig  Hale,  AVISTA,  Inc.  (Part  1,  Part  2,  Handout) 

•  10708  -  Developing  Process  Performance  Models  and  Baselines  for  a  Quality  Engineering  Organization,  Dr.  Jeff  Ricketts,  Raytheon  Company 

TRACK  4  -  HIGHLANDS  -  APPRAISALS 

•  1 1150  -  A  Real-Life  Example  of  Appraising  and  Interpreting  CMMI®  Services  Maturity  Level  2,  Mr.  Neil  Potter,  The  Process  Group 

•  11308  -  Effectively  Managing  Process  Compliance,  Mr.  Joseph  Morin,  Integrated  System  Diagnostics,  Inc. 

•  1 1299  -  Leaner  SCAMPI  Preparation,  Mr.  Gary  Natwick,  Harris  Corporation 
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•  1 1202  -  PHD  Strategies  in  a  Changing  Environment,  Mr.  Sam  Fogle,  ACE  Guides,  LLC 

TRACK  5  -  WIND  RIVER  -  CMMI  V1.3 

•  1 1296  -  CMMI®  VI.  3:  The  Rest  of  the  Story:  Changes  at  CL/ML  1-3,  Dr.  Mike  Konrad,  Software  Engineering  Institute 

Return  to  the  top 


Wednesday,,  November  17f  2010 


Concurrent  Sessions 

TRACK  1  -  CHASM  CREEK-  CMMI®  AND  PROCESS  IMPROVEMENT 

•  11305  -  Turning  a  Super  Tanker:  Process  Improvement  Change  at  NSA,  Mr.  Steven  Tobin,  Dynamics  Research  Corporation 

•  1 1 1 82  -  EPG  -  Pushing  Past  the  Conflicts ,  Ms.  Evelyn  Livermore,  ARA,  Inc. 

•  1 1309  -  Value  Based  Product  Development,  Mr.  Phillip  Hohn,  ITT  ES 

•  1 1204  -  Managing  CMMI  Practice  Implementation  Across  Diverse  Distributed  Environments  in  Small  Organizations,  Dr.  Mary  Anne  Herndon, 
Transdyne  Corporation 

•  1 1209  -  CMMI  Bottoms  -  Ups  Assessment:  A  Grounded  Analysis  from  the  Perspective  of  Practicing  Engineers  in  Defense  Engineering ,  Dr. 
Bruce  Beaded,  General  Dynamics 

•  11339  -  Architecture  for  Enterprise  Process  Improvement ,  Ms.  Joan  Weszka,  Lockheed  Martin  Corporation 

•  1 1221  -  Improvement  by  the  Numbers ,  Ms.  Debra  Perry,  Harris  Corporation 

•  1 1 147  -  Multi-Model  Organizational  Process  Quality  Assurance  Program ,  Ms.  Kristin  Kingland  Glowaski,  Kingland  Systems  Corporation 

TRACK  2  -  WIND  STAR  -  PRACTICAL  GUIDANCE 

•  1 1 161  -  The  Tailoring  Dilemma ,  Mr.  Alan  Gellis,  The  Boeing  Company 

•  1 1270  -  Vertically  Slicing  the  CMMI ,  Mr.  David  Dickinson,  Northrop  Grumman  Corporation 

•  10969  -  CMMI for  Executives ,  NDIA  CMMI  Working  Group 

•  1 1284  -  Measurement  Planning  and  Data  Collection ,  Mr.  William  Golaz,  Lockheed  Martin  Corporation 

•  1 1 3 1 7  -  Experience  Report  On  A  method  To  Predict  Software  Quality  Early  in  the  Software  Development  Lifecycle  and  Produce  Secure 
Software ,  Mr.  Girish  Seshargiri,  AIS 

•  11331  -  A  CMMI  Compliant  Project  Plan  ( In  Less  Than  10  Pages),  Mr.  Ferol  Lewis,  Northrop  Grumman  Corporation 

•  10723  -  Using  CMMI  to  Integrate  and  Improve  the  Enterprise ,  Mr.  Phillip  Wah,  Jr.,  The  Boeing  Company 

•  10779  -  CMMI  -  The  Way  It  Should  Be ,  Mr.  Bradley  Bittorf,  Raytheon  Missile  Systems 

TRACK  3  -  MESA  VERDE  -  PERFORMANCE  RESULTS 

•  10778  -  High  Maturity  Heresy  -  Doing  Level  5  before  Level  4  without  ‘Data  ”,  Mr.  Tom  Lienhard,  Raytheon  Company 

•  10791  -  Why  CMMI  Maturity  Level  5?,  Mr.  Michael  Campo,  Raytheon  Company 

•  10793  -  Horizontal  Segmentation  of  Process  Performance  Model ,  Mr.  Michael  Campo,  Raytheon  Company 

•  10968  -  CMMI  For  Executives ,  NDIA  CMMI  Working  Group 

•  1 1 3 1 6  -  Experience  Report:  Lean  Implementation  of  Organizational  Process  Focus  ( OPF)  and  Risk  Management  (RSKM)  CMMI  Process 
Areas  in  Agile  Environment ,  Mr.  Ahmed  Mahdy,  Raya  Software 

•  10699  -  Improving  Performance  Through  Process  Improvement ,  Mr.  Michael  West,  Natural  SPI,  Inc. 

•  1 1272  -  Get  It  Sold,  Keep  It  Sold:  Making  the  Business  Case  for  High  Maturity ,  Dr.  Richard  Welch,  Northrop  Grumman  Corporation 

•  1 1290  -  Benefits  and  Performance  Improvements  with  CMMI  -  How  to  Increase  Your  Project  Data  Usability ,  Mr.  Kobi  Vider,  K.V.P 
Counsulting 

TRACK  4  -  HIGHLANDS  -  APPRAISALS 

•  1 1 153  -  Appraisals  and  CMMI  Gotchas-  Lessons  in  CMMI  Use  and  Appraisal  Preparations ,  Mr.  Neil  Potter,  The  Process  Group 

•  1 1 1 92  -  Disruptive  Appraisals  Prohibited ,  Ms.  Kelly  Lanier,  Raytheon  Company 

•  1 1265  -  In  Search  of  Higher  Quality  at  Lower  Costs ,  Ms.  Margaret  Corr,  General  Dynamics 

•  1 1321  -  Use  of  a  Standard  Approach  -  Lessons  Learned  in  Identifying  Objective  Evidence  for  the  PHD ,  Ms.  Pamela  Hudson,  Lockheed  Martin 
Corporation 

TRACK  4  -  HIGHLANDS  -  ACQUISITIONS 

•  10550  -  Leveraging  Capability  Maturity  Model  Integration  for  Acquisition  (CMMI-ACQ)  Processes  to  Improve  Organizational  Workforce  and 
System  Acquisition  Performance ,  Dr.  Kenneth  Nidiffer,  Software  Engineering  Instititute 

•  1 1293  -  Using  the  SEI Models  and  Practices  to  Assure  the  System  Lifecycle  Deliverables  Quality  and  Compatibility ,  Mr.  Kobi  Vider,  K.V.P 
Consulting 

•  11328  -  Systems  Asssurance,  The  Global  Supply  Chain,  and  Efforts  To  Increase  Communication  Between  Acquisition  and  Development ,  Ms. 
Michele  Moss,  Booz  Allen  Hamilton 

TRACK  5  -  WIND  RIVER  -  CMMI®  V1.3 

•  1 1 364  -  CMMI  -DEW  1.3  Pilot  -  Good,  Bad  &  Ugly ,  Mr.  Jackie  Lawrence,  ISD,  Inc 

•  11329  -  Tips  and  Hints  for  CMMI  VI.  3,  Ms.  Mary  Beth  Chrissis,  Software  Engineering  Institute 

•  10930  -  CMMI  Version  1.3  High  Maturity  Panel  -  We  Thought  but  Now  We  Know ,  Ms.  Mary  Lynn  Penn,  Lockheed  Martin  Corporation 

•  10986  -  Pilot  for  CMMI  Dev  VI. 3  High  Maturity  Process  Areas ,  Ms.  Doma  Witkowski,  Lockheed  Martin  Corporation 

•  1 07 1 4  -  CMMI  and  Micro  Focus  ’  AVP- A  Dynamic  Duo  in  Optimizing  IT  Business  Value!,  Mr.  Pete  Dupre  ’ ,  MicroF  ocus 
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•  1 121 1  -  The  Perils  of  Agile,  but....,  Ms.  Dorothy  Acton,  Lockheed  Martin  Corporation 
TRACK  5  -  WIND  RIVER  -  CMMI®  AND  AGILE/LEAN 

•  1 1 152  -  Comparing  Scrum  and  CMMI  How  Can  They  Work  Together?,  Mr.  Neil  Potter,  The  Process  Group 

•  1 1 166  -  Practical  Report:  CMMI  Requirements  Development  and  Management  in  an  Agile  Environment ,  Mr.  Ahmed  Mahdy,  Raya  Corporation 

Return  to  the  top 


Thursday,  November  18,  2010 


Concurrent  Sessions 

TRACK  1  -  CHASM  CREEK-  CMMI®  AND  PROCESS  IMPROVEMENT 

•  1 0987  -  Applying  CMMI  -  SVC  to  Process  Management ,  Ms.  Doma  Witkowski,  Lockheed  Martin  Corporation 

•  1 1 141  -  Improving  Operational  Resilence  Processes ,  Mr.  Richard  Barbour,  Software  Engineering  Institute 

•  10701  -  Making  Process  Deployment  and  Monitoring  Work ,  Ms.  Susan  Byrnes,  Natural  SPI,  Inc 

•  1 1 3 1 9  -  Proactive  Process  Deployment  is  Key  to  Institutionalization ,  Ms.  Pamela  Hudson,  Lockheed  Martin 
Corporation 

TRACK  2  -  WIND  STAR  -  PRACTICAL  GUIDANCE 

•  1 1285  -  Deployment  of  Measurement  and  Analysis  at  a  large  Multisite  company ,  Mr.  William  Golaz,  Lockheed 
Martin  Corporation 

•  1 1273  -  Divide  and  Conquer  -  Marshaling  the  “Masses  ”  to  Tackle  CMMI ,  Mr.  Robert  Tuthill,  Northrop 
Grumman  Corporation 

TRACK  3  -  MESA  VERDE  -  PERFORMANCE  RESULTS 

•  1 1292  -  The  Issue  of  Performance  -  Why  Do  You  Need  A  Maturity  Level  5,  Mr.  Kobi  Vider,  K.V.  P  Consulting 

•  1 1 3 1 0  -  Performance  Effects  of  Measurement  and  Analysis ,  Dr.  Dennis  Goldenson,  Software  Engineering 
Institute 

•  1 1 177  -  Connecting  the  Dots:  QPM,  OPP,  CAR  and  OID ,  Ms.  Diane  Mizukami- Williams,  Northrop  Grumman 
Corporation 

TRACK  4  -  HIGHLANDS  -  ACQUISITIONS 

•  10942  -  How  Can  I  Establish  A  Process  Architecture  and  Use  It  Effectively?,  Dr.  Gary  Palosaari,  The  Boeing 
Company 

•  How  MBE  can  support  Requirements  Development  and  Technical  Solution ,  Mr.  Frank  Salvatore,  High 

Performance  Technologies,  Inc 

•  11165  —  A  Streamlined  Approach  to  SW  Estimating  Using  Cocomo  If  Mr.  John  Rossi,  Rockwell  Collins 

•  10599  -  Systems  Engineering  Processes  Improvement  Using  the  CMMI  in  Large  System  of  Systems  Space 
Programs ,  Ms.  Sarit  Assaraf,  Israel  Aerospace  Industries 

TRACK  5  -  WIND  RIVER  -  CMMI®  AND  AGILE/LEAN 

•  1 12 1 3  -  Using  the  Agile  Methodology  to  Mitigate  the  Risks  of  Highly  Adaptive  Projects ,  Ms.  Dana  Roberson, 
Transdyne  Corporation 

•  11323  -  Lean  Development  -  Influencing  the  Organization ,  Mr.  Carl  Gaul,  The  Boeing  Company 

•  11324  -  Lean  Development  -  Selecting  the  Right  Tools  for  Maximum  Success,  Mr.  Carl  Gaul,  The  Boeing 
Company 
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National  Defense  Industrial  Association 

PROMOTING  NATIONAL  SECURITY  SINCE  1919 


1  0th  annual 

CMMI®  TECHNOLOGY 
CONFERENCE  AND  USER  GROUP 


CMMI® 


Software  Engineering  Institute  CarnegieMellon 


SPONSORED  BY: 

NATIONAL  DEFENSE  INDUSTRIAL 
ASSOCIATION  SYSTEMS 
ENGINEERING  DIVISION 
IN  CONJUNCTION  WITH: 

Software  Engineering  Institute 

Carnegie  Mellon 

This  conference  brings  together  the  managers 
and  professionals  involved  in  Acquisition 
Management,  Systems  Engineering,  Program 
Management,  Software  Development,  Process 
Improvement,  Six  Sigma  and  related  activities  for 
the  purpose  of  advancing  the  state-of-the-art  in 
process  improvement  and  achieving  a  higher  state 
of  process  capability  in  engineering  development 
in  order  to  reduce  cost,  schedule  and  risk,  and 
improve  overall  quality. 

Who  Should  Attend? 

Defense,  aerospace  and  commercial  companies, 
CMMI®  Transition  Partners,  Department 
of  Defense  organizations,  small  companies 
specializing  in  software  and  systems  engineering 
development,  tools  and  processes,  acquisition,  or 
services,  and  other  government  agencies. 

What  Will  Be  Presented? 

A  wide  variety  of  presentations,  including  the 
new  CMMI®  for  Services,  integrated  process 
improvement,  Lean/Agile  and  Six  Sigma 
approaches,  and  evolving  approaches  and  lessons 
learned  involving  SCAMPISM  appraisal  methods. 
The  latest  state  of  the  CMMI®  V.3  release  will  be 
presented  and  questions  answered. 


CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 


CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 

HYATT  REGENCY  TECH  CENTER  ►  DENVER,  COLORADO 
NOVEMBER  15-18, 2010 

The  CMMI®  was  developed  in  cooperation  with  the  Department  of  Defense,  Industry  and 
the  Software  Engineering  Institute,  and  has  become  the  defacto  standard  for  integrated 
process  improvement  across  multiple  disciplines  within  commercial,  Department  of 
Defense  and  government  organizations.  Although  sponsored  by  the  Department  of 
Defense  and  NDIA,  the  CMMI®  is  used  by  commercial  as  well  as  government  and 
industry  organizations,  and  this  conference  will  address  all  applications.  The  purpose 
of  CMMI®  is  to  provide  for  improvements  in  cost,  schedule  and  overall  performance  of 
projects  in  engineering,  acquisition,  and  services  by  eliminating  “stovepipe”  maturity 
models  and  allowing  organizations  to  integrate  their  process  improvement  efforts. 
CMMI®  has  been  shown  to  reduce  costs,  to  implement  internal  process  improvement, 
including  appraisals  (assessments  and  evaluations)  and  provide  a  common  baseline  and 
lexicon  for  process  improvement. 

LODGING 

A  block  of  rooms  has  been  reserved  at  the  Hyatt  Regency  Denver  Tech  Center.  The 
government  per  diem  rate  is  $  14 1  U.S.  (*Or  the  prevailing  government  per  diem  at  the 
time  of  the  conference).  The  industry  rate  is  $167.00  US. 

In  order  to  ensure  the  discounted  NDIA  rate,  please  make  reservations  early  and  ask  for 
the  NDIA  room  block.  Rooms  will  not  be  held  after  October  23,  2010  and  may  also 
sell  out  before  then.  Rates  are  subject  to  increase  after  this  date. 

*The  per  diem  rate  is  available  only  to  active  duty  or  civilian  government  employees. 
ID  will  be  required  upon  check-in.  Retired  military  or  government  civilians  do  not 
qualify  for  the  government  rate. 

Government  Attendees:  Any  active  U.S.  Military  attendee  with  ID  and  travel 
order  will  be  tax  exempt  (subject  to  states  regulation  of  tax  exemption).  Any  U.S. 
Government  employee  paying  with  U.S.  Government  check  or  credit  card  will  be  tax 
exempt  (with  supporting  documentation  as  required  by  some  states).  Check  with  the 
hotel  for  specific  state  and  local  requirements  when  booking  hotel  reservations. 

SPECIAL  NEEDS 

NDIA  supports  the  Americans  with  Disabilities  Act  of  1990.  Attendees  with  special 
needs  should  call  Kelly  Seymour  at  (703)  247-2583  prior  to  November  5,  2010. 

CONFERENCE  ATTIRE 

Appropriate  dress  for  this  conference  is  business  for  civilians  (coat  and  tie)  and  class  A 
uniform  or  uniform  of  the  day  for  military. 


ANNOUNCEMENT 

The  National  Defense  Industrial 
Association,  Systems  Engineering 
Division,  in  conjunction  with  the 
Software  Engineering  Institute,  Carnegie 
Mellon  University,  is  pleased  to  announce 
the  1 0th  Annual  CMMI®  (Capability 
Maturity  Model  Integration)  Technology 
Conference  &  User  Group.  This  premier 
conference  will  be  held  November  15-18, 
2010,  at  the  Hyatt  Regency  Tech  Center 
in  Denver,  Colorado. 

The  purpose  of  the  conference  is  to 
exchange  ideas,  concepts  and  lessons 
learned  concerning  the  continuing 
evolution,  adoption  and  use  of  the 
CMMI®  and  its  associated  appraisal 
(assessment  and  evaluation)  methods. 

This  conference  brings  together  CMMI® 
adopters,  users,  developers  and  appraisers, 
as  well  as  those  with  general  interest 
in  process  improvement.  It  provides  a 
forum  for  the  free  exchange  of  ideas  and 
affords  a  unique  opportunity  to  meet  with 
the  sponsors,  developers  and  stewards 
of  the  CMMI®,  as  well  as  those  offering 
CMMI®  training  and  implementation 
assistance.  Emphasis  will  be  placed  on 
CMMI®  implementation  methods  and 
strategies,  return  on  investment  and 
program/project  performance  benefits. 

CONTACTS 

Ms.  Kelly  Seymour,  Meeting  Planner, 
kseymour@ndia.org,  (703)  247-2583 

Conference  Chair:  Mr.  Bob  Rassa, 
Director,  Systems  Support,  Raytheon 
Company,  rcrassa@raytheon.com,  (310) 
985-4962 

Technical  Program  Chairs:  Mr.  Rick 
Barbour,  Software  Engineering  Institute, 
reb@sei.cmu.edu;  Brian  Gallagher, 
Northrop  Grumman  Corporation,  Brian. 
Gallagher@ngc.com 


CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 


PROMOTIONAL 

PARTNERSHIPS 

Increase  your  company  or  organizations 
exposure  at  this  premier  event  by 
becoming  a  Promotional  Partner.  A 
Promotional  Partnership  ($5,000)  will 
add  your  company  name  to  the  back 
cover  of  the  on-site  brochure  as  well  as 
main  podium  recognition  throughout  the 
conference,  signage  at  all  events  including 
the  opening  reception,  and  a  350-word 
organization  description  in  the  conference 
agenda.  For  more  information,  please 
contact  Sam  Campagna  at  703-247-2544 
or  scampagna@ndia.org  or  Kelly  Seymour 
at  703-247-2583  or  kseymour@ndia.org. 

DISPLAYS 

An  area  will  be  available  for  the  setup 
of  company  or  organization  displays  to 
demonstrate  unique  tools  or  capability 
in  process  improvement,  transition 
partner  activities,  software  and  systems 
engineering  tools  and  related  capability. 

The  registration  fee  of  $1,800  includes 
one  individual  registration,  table,  chair 
and  electrical  hook-up.  Displays  will  be 
tabletop  or  “pop-up”  style.  Allocated 
display  space  will  be  1 0  ft.  wide  by 
6  ft.  deep.  Pipe  and  drape  will  be 
provided.  Although  computer  systems 
for  demonstration  are  allowed,  minimal 
hardware  should  be  utilized.  No  formal 
decorating  company  will  be  involved. 
Companies  must  bring  their  own  displays 
and  plan  to  do  their  own  set-up.  Standard 
draped  folding  table  and  chair  will  be 
provided  for  each  display  space. 

Display  setup  will  begin  Monday, 
November  15  at  7:00  am  in  the  Second 
Floor  Atrium  of  the  Hyatt  Regency 
Denver  Tech  Center.  Display  location 
is  on  a  first-come,  first-served  basis  and 
the  locations  are  not  assigned  ahead  of 
time.  Displays  are  to  be  set  by  4:00  pm 
on  Monday,  November  1 5  and  must 
remain  in  place  until  the  morning  break 
on  Thursday,  November  18.  Displays 
must  be  removed  by  2:00  pm  on  Thursday, 
November  18. 


MONDAY,  NOVEMBER  15, 2010 
TUTORIAL  REGISTRATION 

Tutorials  provide  a  broad  range  of  informative  topics  that  will  enable  a  better 
understanding  and  usage  of  elements  of  the  CMMI®  product  suite.  This  includes 
understanding  the  CMMI®-ACQ  and  CMMP-SVC  as  well  as  practical  guidance  and 
experience  in  applying  the  CMMI®  model.  These  tutorials  also  include  experience  in 
incorporating  supporting  techniques  such  as  lean  methods  and  processes,  and  EVM. 


TUTORIAL 

REGISTRATION  FEES 

REGULAR 
(BEFORE  11/05) 

LATE 

(AFTER  11/5) 

INDUSTRY/GOVERNMENT 

$250 

$275 

TUESDAY,  NOVEMBER  16  - 
THURSDAY,  NOVEMBER  18 
CMMI®  CONFERENCE  REGISTRATION 

Register  online  by  visiting  the  conference  website  at  www.ndia.org/meetings/ 1 1 10. 
Online  registration  will  close  at  5:00  pm  EST  on  November  5,  2010.  You  may  also 
fax  the  registration  form  found  in  this  brochure  to  (703)  522-1885  or  mail  it  to 
National  Defense  Industrial  Association,  Event  #1110,  2111  Wilson  Blvd.,  Suite 
400,  Arlington,  VA,  22201.  Payment  must  be  made  at  the  time  of  registration. 
Registrations  will  not  be  taken  over  the  phone. 


CONFERENCE 
REGISTRATION  FEES 

EARLY 

(ON/BEFORE  10/01) 

REGULAR 

(10/02-11/05) 

LATE 

(AFTER  11/05) 

GOVERNMENT/ 
ACADEMIA/ALLIED  GOV. 

$750 

$825 

$930 

INDUSTRY 

NDIA  MEMBER 

$850 

$935 

$1050 

INDUSTRY 

N0N-NDIA  MEMBER 

$970 

$1070 

$1180 

Cancellations  received  on  or  before  11/5/10  will  receive  a  refund  minus  a  $75 
cancellation  fee.  No  refunds  will  be  given  for  cancellations  after  11/5/10.  Substitutions 
are  welcome  in  lieu  of  cancellations.  Cancellations  and  substitutions  must  be  made  in 
writing  to  kseymour@ndia.org. 


Visit  www.ndia.org/meetings/ 1 1 10  to 
download  the  Display  Registration  Form. 
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SUNDAY,  NOVEMBER  14, 2010 

4:00  pm  -  6:00  pm  Registration  Open 

Located  in  Mesa  Verde  Foyer,  1st  Floor 

MONDAY,  NOVEMBER  15, 2010 

7:00  am  -  6:00  pm  Registration  Open 

Located  in  Mesa  Verde  Foyer,  1st  Floor 

7:00  am  -  8:00  am  Continental  Breakfast  (Tutorial  Attendees  Only) 

Located  in  Wind  Star  Foyer,  1st  Floor 

8:00  am  -  5:00  pm  Tutorial  Sessions  (Tutorial  Attendees  Only;  Additional  Cost 
Required) 

9:45  am  - 1 0:1 5  am  Break  (Tutorial  Attendees  Only) 

Located  in  Wind  Star  Foyer,  1st  Floor 

1 1 :45  am  - 1 :00  pm  Lunch  (Tutorial  Attendees  Only) 

Located  in  Centennial,  12th  Floor 

2:45  pm  -  3:1 5  pm  Break  (Tutorial  Attendees  Only) 

Located  in  Wind  Star  Foyer,  1st  Floor 

5:00  pm  -  6:00  pm  Reception  (Open  to  ALL  ATTENDEES) 

Located  in  Atrium  Display  Area,  2nd  Floor 

TUESDAY,  NOVEMBER  16, 2010 

7:00  am  -  6:00  pm  Registration  Open 

Located  in  Mesa  Verde  Foyer,  1st  Floor 

7:00  am  -  8:00  am  Continental  Breakfast 

Located  in  Atrium  Display  Area,  2nd  Floor 

8:00  am  -  8:15  am  Welcome  and  Opening  Remarks 

Located  in  Grand  Mesa  ABC,  2nd  Floor 

►  Mr.  Sam  Campagna,  Assistant  Vice  President,  Operations,  NDIA 

►  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Company 

8:1 5  am  -  9:00  am  Keynote 

Located  in  Grand  Mesa  ABC,  2nd  Floor 

►  Mr.  Terence  Edwards,  Director,  System- of- Systems  Engineering  (SoSE), 
ASA(ALT) 

9:00  am  - 10:15  am  CMMI®  Implementation  Strategies  for  Sucess  Panel 

Located  in  Grand  Mesa  ABC,  2nd  Floor 

Moderator:  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon 
Company 

►  Ms.  Nancy  Fleishcher,  Raytheon  Company 

►  Ms.  Juliet  Davis,  The  Boeing  Company 

►  Mr.  Paul  Zavidniak,  Northrop  Grumman  Corporation 

►  Ms.  Bettina  Schechter,  Lockheed  Martin  Mission  Systems  &  Sensors  (MS2) 

10:15  am -10:30  am  Break 

Located  in  Atrium  Display  Area,  2nd  Floor 

10:30  am  -  11:15  am  CMMI®  VI  .3  Overview 

►  Mr.  Mike  Phillips,  Software  Engineering  Institute 


11:15  am -12:10  pm  BENEFITS  OF  CMMI® 

Located  in  Grand  Mesa  ABC,  2"d  Floor 

►  Mr.  Hal  Wilson,  Director,  Engineering,  Defense  Systems  Division, 
Northrop  Grumman 

►  Mr.  Bob  Rassa,  Director,  Engineering  Programs,  Raytheon  Company 

12:10  pm -12:30  pm  Q&A 

12:30  pm -1:30  pm  Lunch 

Located  in  Grand  Mesa  F,  2nd  Floor 

1 :30  pm  -  3:00  pm  Concurrent  Sessions 

3:00  pm  -  3:30  pm  Break 

Located  in  Atrium  Display  Area,  2nd  Floor 

3:30  pm  -  5:00  pm  Concurrent  Sessions 


5:30  pm  -  6:30  pm 


Reception 

Located  in  Atrium  Display  Area,  2nd  Floor 


WEDNESDAY,  NOVEMBER  17, 2010 


7:00  am  -  4:30  pm 

7:00  am  -  8:00  am 

8:00  am  -  9:30  am 
9:30  am  - 10:00  am 

10:00  am  - 11:30  am 
11:30  am -1:00  pm 

1:00  pm -2:30  pm 
2:30  pm  -  3:00  pm 

3:00  pm  -  4:30  pm 
4:30  pm 


Registration  Open 

Located  in  Mesa  Verde  Foyer,  1st  Floor 

Continental  Breakfast 

Located  in  Atrium  Display  Area,  2nd  Floor 

Concurrent  Sessions 
Break 

Located  in  Atrium  Display  Area,  2nd  Floor 

Concurrent  Sessions 
Awards  Lunch 

Located  in  Grand  Mesa  F,  2"d  Floor 

Concurrent  Sessions 
Break 

Located  in  Atrium  Display  Area,  2nd  Floor 

Concurrent  Sessions 
Conference  adjourns  for  the  day 


THURSDAY,  NOVEMBER  18, 2010 

7:00  am  - 1 1 :30  am  Registration  Open 

Located  in  Grand  Mesa  Foyer,  2nd  Floor 

7:00  am  -  8:00  am  Continental  Breakfast 

Located  in  Atrium  Display  Area,  2nd  Floor 

8:00  am  -  9:30  am  Concurrent  Sessions 

9:30  am  - 1 0:00  am  Break 

Located  in  Atrium  Display  Area,  2nd  Floor 

10:00  am  - 11:30  am  Concurrent  Sessions 
1 1 :30  am  -  2:00  pm  Display  Dismantle 
1 1 :30  am  Conference  adjourns 


**Please  note  that  the  agenda  is  subject  to  change** 
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AND  USER  GROUP 


MONDAY,  NOVEMBER  15, 2010 


Track  1 

Track  2 

Track  3 

Track  4 

Track  5 

Chasm  Creek 

Wind  Star 

Mesa  Verde 

Highlands 

Wind  River 

8:00  am  - 

1A1  -  Tutorial 

1A2  -  Tutorial 

1A3  -  Tutorial 

1A4  -  Tutorial 

1A5  -  Tutorial 

9:45  am 

10924  -CMMI®  VI  .3— 

10960  -Risk 

11151  -  Making  Process 

11274  -  Applying  Lean 

11262  -SPI  Manifesto 

Session  A 

What’s  New? 

Management  and  Beyond 

Improvement  Work  — 

Principles  to  CMMI® 

—  Values  and  Principles 

Tying  Improvement  and 

Implementations  to 

CMMI®  Directly  to  What 
You  Care  About 

Achieve  More  R0I 

Mr.  Mike  Phillips, 

Software  Engineering 

Mr  Al  Florence,  The 

Mr.  Neil  Potter,  The 

Mr.  Tim  Olson,  Lean 

Mr.  Tim  Kasse,  Kasse 

Institute 

MITRE  Corporation 

Process  Group 

Solutions  Institute,  Inc. 

Initiatives,  LLC 

10:15  am- 

1B1  -  Tutorial  (Con’t..) 

1B2  -  Tutorial  (Con’t..) 

1B3  -  Tutorial  (Con’t..) 

1B4  -  Tutorial  (Con’t..) 

1B5  -  Tutorial  (Con’t..) 

11:45  am 

10924  -CMMI®  VI  .3— 

10960  -Risk 

11151  -  Making  Process 

11274  -  Applying  Lean 

11262  -SPI  Manifesto 

Session  B 

What’s  New? 

Management  and  Beyond 

Improvement  Work  — 

Principles  to  CMMI® 

—  Values  and  Principles 

Tying  Improvement  and 

Implementations  to 

CMMI®  Directly  to  What 
You  Care  About 

Achieve  More  R0I 

Mr  Mike  Phillips, 

Software  Engineering 

Mr  Al  Florence,  The 

Mr.  Neil  Potter,  The 

Mr.  Tim  Olson,  Lean 

Mr.  Tim  Kasse,  Kasse 

Institute 

MITRE  Corporation 

Process  Group 

Solutions  Institute,  Inc. 

Initiatives,  LLC 

Break  in  Wind  Star  Foyer,  1st  Floor 

1 :00  pm  - 

1C1  -  Tutorial 

1 C2  -  Tutorial 

1 C3  -  Tutorial 

1 C5  -  Tutorial 

2:45  pm 

11203-  CMMI®  VI  .3 

1 1 288  -  Strategic 

10962  -Life  Cycle 

1 1 263  -  Effective 

and  Architecture 

Technology  and 

Configuration 

Technology  Transition 

Session  C 

Operational  Risk 

Management 

Techniques  That  Make 

Management  (STORM) 

Process  Improvement 
Happen 

Dr  Lawrence  Jones, 
Software  Engineering 

Mr.KobiVider,  K.V.P 

Mr.  Al  Florence,  The 

Mr.  Tim  Kasse,  Kasse 

Institute 

Consulting 

MITRE  Corporation 

Initiatives,  LLC 

3:15  pm  - 

1D1  -  Tutorial  (Con’t..) 

1D2  -  Tutorial  (Con’t..) 

1D3  -  Tutorial  (Con’t..) 

1D5  -  Tutorial  (Con’t..) 

5:00  pm 

11203-  CMMI®  VI  .3 

1 1 288  -  Strategic 

10962  -Life  Cycle 

1 1 263  -  Effective 

and  Architecture 

Technology  and 

Configuration 

Technology  Transition 

Session  D 

Operational  Risk 

Management 

Techniques  That  Make 

Management  (STORM) 

Process  Improvement 
Happen 

Dr  Lawrence  Jones, 
Software  Engineering 

Mr.KobiVider,  K.V.P 

Mr.  Al  Florence,  The 

Mr.  Tim  Kasse,  Kasse 

Institute 

Consulting 

MITRE  Corporation 

Initiatives,  LLC 

**Additional  Cost  Required** 
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AND  USER  GROUP 


TUESDAY,  NOVEMBER  16, 2010 


Track  1 

Track  2 

Track  3 

Track  4 

Track  5 

Chasm  Creek 

Wind  Star 

Mesa  Verde 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Performance  Results 

Appraisals 

CMMI®  VI  .3 

Session  Chairs: 

Ms.  Susan  Bassham, 
U.S.  Army 

Session  Chairs: 

Mr.  Geoff  Draper,  Harris 
Corporation,  Ms.  Diane 
Mizukami  Williams, 
Northrop  Grumman 
Corporation 

Session  Chair: 

Mr.  Dennis  Goldenson, 
Software  Engineering 
Institute,  Mr.  Ken 
Weinberg,  Raytheon 
Company 

Session  Chairs: 

Ms.  Lorraine  Adams, 
Software  Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software  Engineering 
Institute,  Mr.  Charles 
Ryan,  Software 
Engineering  Instititute 

1:30  pm- 
2:15  pm 

11 31 4 -CMMI®  in  the 
Social  Media  (For  the 

Social  Media-Challenged! 

Mr.  Bill  Smith,  Leading 

Edge  Process  Consultants, 
LLC 

10963  -  Configuration 
Management 

Mr  Al  Florence,  The 

MITRE  Corporation 

11 307 -Process 
Performance  Models  for 
Hardware  Engineers 

Mr.  Tom  Fosness,  Northrop 
Grumman  Corporation 

11150 -A  Real-Life 
Example  of  Appraising 
and  Interpreting  CMMI® 
— Services  Maturity 

Level  2 

Mr.  Neil  Potter,  The 

Process  Group 

11 296  -CMMI®  VI  .3: 

The  Rest  of  the  Story: 
Changes  at  CL/ML  1  -3 

Dr  Mike  Konrad, 

Software  Engineering 
Institute 

2:15  pm  - 
3:00  pm 

11 31 5 -What  Doesn’t  Kill 
You  Makes  You  Stronger: 

My  Process  Improvement 
Lessons  Learned 

10970  -Using  CMMI® 
Effectively  in  Small 
Business 

11185  -  Navigating 
the  Waters  of  SCAMPI 
Reappraisal 

1 1 308  -  Effectively 
Managing  Process 
Compliance 

1 1 327  -  Questions 
and  Answers  with  the 
SCAMPI  Upgrade  Team 

Mr  Bill  Smith,  Leading 

Edge  Process  Consultants, 
LLC 

NDIA  CMMI®  Working 
Group 

Mr.  Craig  Hale,  AVISTA, 

Inc. 

Mr.  Joseph  Morin, 
Integrated  System 
Diagnostics,  Inc. 

Mr.  Will  Hayes,  Software 
Engineering  Institute 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Performance  Results 

Appraisals 

CMMI®  VI  .3 

Session  Chairs: 

Ms.  Susan  Bassham, 
U.S.  Army 

Session  Chairs: 

Mr.  Geoff  Draper,  Harris 
Corporation,  Ms.  Diane 
Mizukami  Williams, 
Northrop  Grumman 
Corporation 

Session  Chair: 

Mr.  Dennis  Goldenson, 
Software  Engineering 
Institute,  Mr.  Ken 
Weinberg,  Raytheon 
Company 

Session  Chairs: 

Ms.  Lorraine  Adams, 
Software  Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software  Engineering 
Institute 

3:30  pm  - 
4:15  pm 

11 31 8 -NASA’s  Use  of 
the  CMMI®  for  Software 
Engineering  Improvement 

Dr  John  Kelly,  NASA 

11 340  -Risk 

Management  -  How 

QPM  Saved  the  Day 

Ms.  Deepti  Sharma,  OST 

11338  -  From  Denial 
to  Commitment — 
Quantitative  Chronicles  of 
an  Organization’s  Cultural 
Change  During  its 

Journey  Towards  Higher 
Maturity 

Ms.  Kinkini  Sarkar,  OST 

11 299  -Leaner  SCAMPI 
Preparation 

Mr.  Gary  Natwick,  Harris 
Corporation 

11334  -  Scoping 
Appraisals  Using  SCAMPI 
VI  .3 

Mr.  Will  Hayes,  Software 
Engineering  Institute 

4:15  pm  - 
5:00  pm 

11297  -  Applying  Theory 
of  Constraints  Tools  to 
Focus  Lean  Development 

1 1 1 58  -  Process 
Alignment — From 

Acquired  to  Acquirer 

10708  -  Developing 
Process  Performance 
Models  and  Baselines 
for  a  Quality  Engineering 
Organization 

11202  -  PHD  Strategies 
in  a  Changing 

Environment 

11 335  -Using  SCAMPI 
with  More  than  One 
CMMI®  Constellation  in 
the  Same  Appraisal 

Mr  Tim  Oltman,  The 

Boeing  Company 

Ms.  Ilene  Pinsker, 

Rockwell  Collins 

Dr.  Jeff  Ricketts, 

Raytheon  Company 

Mr.  Sam  Fogle,  ACE 
Guides,  LLC 

Mr.  Will  Hayes,  Software 
Engineering  Institute 

CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 


WEDNESDAY,  NOVEMDER 17, 2010 


Track  1 

Track  2 

Track  3 

Track  4 

Track  5 

Chasm  Creek 

Wind  Star 

Mesa  Verde 

Highlands 

Wind  River 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Performance  Results 

Appraisals 

CMM®  VI  .3 

Session  Chairs: 

Ms.  Susan  Bassham, 
U.S.  Army 

Session  Chairs: 

Mr.  Geoff  Draper,  Harris 
Corporation,  Ms.  Diane 
Mizukami  Williams, 
Northrop  Grumman 
Corporation 

Session  Chair: 

Mr.  Dennis  Goldenson, 
Software  Engineering 
Institute,  Mr.  Ken 
Weinberg,  Raytheon 
Company 

Session  Chairs: 

Ms.  Lorraine  Adams, 
Software  Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software  Engineering 
Institute,  Mr.  Charles 
Ryan,  Software 
Engineering  Instititute 

8:00  am  - 
8:45  am 

11305  -  Turning  a 

Super  Tanker:  Process 
Improvement  Change 
at  NSA 

Mr.  Steven  Tobin, 

Dynamics  Research 
Corporation 

11161  -  The  Tailoring 
Dilemma 

Mr.  Alan  Gellis,  The 

Boeing  Company 

10778  -  High  Maturity 
Heresy — Doing  Level  5 
before  Level  4  without 
“Data” 

Mr.  Tom  Lienhard, 
Raytheon  Company 

1 1 1 53  -  Appraisals  and 
CMM®  Gotchas — 
Lessons  in  CMM® 

Use  and  Appraisal 
Preparations 

Mr.  Neil  Potter,  The 
Process  Group 

11 364 -CMM® -DEV 

VI  .3  Pilot — Good,  Bad 
&  Ugly 

Mr.  Jackie  Lawrence, 

ISD,  Inc. 

8:45  am  - 
9:30  am 

11182 -EPG- Pushing 
Past  the  Conflicts 

11270  -  Vertically  Slicing 
the  CMM® 

10791  -  Why  CMM® 
Maturity  Level  5? 

11192  -  Disruptive 
Appraisals  Prohibited 

11329  -  Tips  and  Hints 
for  CMM®  VI  .3 

Ms.  Evelyn  Livermore, 

ARA,  Inc. 

Mr.  David  Dickinson 
Northrop  Grumman 
Corporation 

Mr.  Michael  Campo, 
Raytheon  Company 

Ms.  Kelly  Lanier, 

Raytheon  Company 

Mrs.  Mary  Beth  Chrissis, 
Software  Engineering 
Institute 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 
Improvement 

Practical  Guidance 

Performance  Results 

Appraisals 

CMM®  VI  .3 

Session  Chairs: 

Ms.  Susan  Bassham, 
U.S.  Army 

Session  Chairs: 

Mr.  Geoff  Draper,  Harris 
Corporation,  Ms.  Diane 
Mizukami  Williams, 
Northrop  Grumman 
Corporation 

Session  Chair: 

Mr.  Dennis  Goldenson, 
Software  Engineering 
Institute,  Mr.  Ken 
Weinberg,  Raytheon 
Company 

Session  Chairs: 

Ms.  Lorraine  Adams, 
Software  Engineering 
Institute  &  Mr.  Mike 
Phillips,  Software 
Engineering  Institute 

Session  Chair: 

Mr.  Ken  Nidiffer, 
Software  Engineering 
Institute,  Mr.  Charles 
Ryan,  Software 
Engineering  Instititute 

10:00  am- 
10:45  am 

11309  -  Value  Based 
Product  Development 

10969  -CMM®  for 
Executives 

1 0793  -  Horizontal 
Segmentation  of  Process 
Performance  Model 

1 1 265  -  In  Search  of 
Higher  Quality  at  Lower 
Costs 

10930  -  CMM®  Version 

1 .3  High  Maturity  Panel 
— We  Thought  but  Now 

We  Know 

Mr.  Phillip  Hohn,  ITTES 

NDIA  CMMI®  Working 
Group 

Mr.  Michael  Campo, 
Raytheon  Company 

Mrs.  Margaret  Corr, 
General  Dynamics 

Ms.  Mary  Lynn  Penn, 
Lockheed  Martin 
Corporation 

10:45  am  - 
11:30  am 

11204  -  Managing 

CMM®  Practice 
Implementation  Across 
Diverse  Distributed 
Environments  in  Small 
Organizations 

Dr  Mary  Anne  Herndon, 
Transdyne  Corporation 

1 1 284  -  Measurement 
Planning  and  Data 
Collection 

Mr.  William  Golaz, 
Lockheed  Martin 
Corporation 

10968  -CMM®  for 
Executives 

NDIA  CMM®  Working 
Group 

11321  -  Use  of  a 

Standard  Approach — 
Lessons  Learned  in 
Identifying  Objective 
Evidence  for  the  PHD 

Ms.  Pamela  Hudson, 
Lockheed  Martin 
Corporation 

10986  -  Pilot  for 
CMM®-DEV  VI  .3  High 
Maturity  Process  Areas 

Ms.  Dorna  Witkowski, 
Lockheed  Martin 
Corporation 

CMMI®  TECHNOLOGY  CONFERENCE 
AND  USER  GROUP 


Track  1 

Track  2 

Track  3 

Track  4 

Track  5 

Chasm  Creek 

Wind  Star 

Mesa  Verde 

Highlands 

Wind  River 

CMMI®  and  Process 

Practical  Guidance 

Performance  Results 

Acquisition 

CMMI®  and 

Improvement 

Agile/Lean 

Session  Chairs: 

Session  Chair: 

Session  Chairs: 

Session  Chair: 

Mr.  Geoff  Draper,  Harris 

Mr.  Dennis  Goldenson, 

Ms.  Lorraine  Adams, 

Mr.  Ken  Nidiffer, 

Corporation,  Ms.  Diane 

Software  Engineering 

Software  Engineering 

Software  Engineering 

Session  Chairs: 

Mizukami  Williams, 

Institute,  Mr.  Ken 

Institute  &  Mr.  Mike 

Institute,  Mr.  Charles 

Ms.  Susan  Bassham, 

Northrop  Grumman 

Weinberg,  Raytheon 

Phillips,  Software 

Ryan,  Software 

U.S.  Army 

Corporation 

Company 

Engineering  Institute 

Engineering  Instititute 

1 :00  pm  - 

11 209 -CMMI® 

1 1 31 7  -  Experience 

1 1 31 6  -  Experience 

10550  -  Leveraging 

10714  -  CMMI  and  Micro 

1 :45  pm 

Bottoms -Up  Assessment: 

Report  On  A  Method  To 

Report:  Lean 

Capability  Maturity  Model 

Focus’ AVP- A  Dynamic 

A  Grounded  Analysis 

Predict  Software  Quality 

Implementation  of 

Integration  for  Acquisition 

Duo  in  Optimizing  IT 

from  the  Perspective  of 

Early  in  the  Software 

Organizational  Process 

(CMMI®-ACQ)  Processes 

Business  Value! 

Practicing  Engineers  in 

Development  Lifecycle 

Focus  (0PF)  and  Risk 

to  Improve  Organizational 

Defense  Engineering 

and  Produce  Secure 

Management  (RSKM) 

Workforce  and  System 

Software 

CMMI®  Process  Areas  in 

Acquisition  Performance 

Agile  Environment 

Dr  Kenneth  Nidiffer, 

Dr  Bruce  Beadell,  General 

Mr.  Ahmed  Mahdy,  Raya 

Software  Engineering 

Mr.  Pete  DuPre’, 

Dynamics 

Mr.  Girish  Seshagiri,  AIS 

Software 

Institute 

MicroFocus 

1 :45  pm  - 

11339  -  Architecture 

11331  -  A  CMMI®- 

10699  -  Improving 

11 293  -Using  the  SEI 

11211  -  The  Perils  of 

2:30  pm 

for  Enterprise  Process 

Compliant  Project  Plan  (in 

Performance  Through 

Models  and  Practices 

“Agile,  but...” 

Improvement 

Less  Than  1 0  Pages 

Process  Improvement 

to  Assure  the  System 
Lifecycle  Deliverables 
Quality  and  Compatibility 

Ms.  Joan  Weszka, 

Mrs.  Dorothy  Acton, 

Lockheed  Martin 

Mr.  Ferol  Lewis,  Northrop 

Mr.  Michael  West,  Natural 

Mr.  Kobi  Vider,  K.  V.P 

Lockheed  Martin 

Corporation 

Grumman  Corporation 

SPI,  Inc. 

Consulting 

Corporation 

Break  in  Atrium  Display  Area,  2nd  Floor 

CMMI®  and  Process 

Practical  Guidance 

Performance  Results 

Acquisition 

CMMI®  and 

Improvement 

Agile/Lean 

Session  Chairs: 

Session  Chair: 

Session  Chairs: 

Session  Chair: 

Mr.  Geoff  Draper,  Harris 

Mr.  Dennis  Goldenson, 

Ms.  Lorraine  Adams, 

Mr.  Ken  Nidiffer, 

Corporation,  Ms.  Diane 

Software  Engineering 

Software  Engineering 

Software  Engineering 

Session  Chairs: 

Mizukami  Williams, 

Institute,  Mr.  Ken 

Institute  &  Mr.  Mike 

Institute,  Mr.  Charles 

Ms.  Susan  Bassham, 

Northrop  Grumman 

Weinberg,  Raytheon 

Phillips,  Software 

Ryan,  Software 

U.S.  Army 

Corporation 

Company 

Engineering  Institute 

Engineering  Instititute 

3:00  pm  - 

11221  -  Improvement  by 

10723  -Using  CMMI® 

11 272  -Get  It  Sold, 

11372  -  Maritime 

11152  -  Comparing 
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Purpose 


This  presentation  proposes  an  approach 
for  supporting  successful  deployment  and 
monitoring  of  process  improvements. 
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Process  Deployment  and  Monitoring 


A  closer  look  at  OPF  SG  3 


Organizational  Process  Focus  (OPF)  Specific  Goal  3 
(SG3)  Deploy  Organizational  Process  Assets  and 
Incorporate  Lessons  Learned 

□  The  organizational  process  assets  are  deployed  across 
the  organization  and  process-related  experiences  are 
incorporated  into  the  organizational  process  assets. 


What  does  "Deployed"  mean  in  your  organization? 
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What  does  this  mean?., 

OPF  SP  3.1  Deploy  Organizational  Process  Assets 

□  Deploy  organizational  process  assets  across  the 
organization. 


OPF  SP  3.1  is  about 
identifying  the  strategy  (and 
the  necessary  support 
resources)  for  deploying  new 
and  changed  assets  to  the 
entire  organization. 
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What  does  this  mean?. 


OPF  SP  3.2  Deploy  Standard  Processes 


□  Deploy  the  organization’s  set  of  standard  processes  to 
projects  at  their  startup  and  deploy  changes  to  them  as 
appropriate  throughout  the  life  of  each  project 


OPF  SP3.2  is  about 
helping  individual  projects 
adopt  and  tailor  current 
organizational  processes 
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What  does  this  mean?3 


OPF  SP  3.3  Monitor  Implementation 

□  Monitor  the  implementation  of  the  organization’s  set  of 
standard  processes  and  use  of  process  assets  on  all 
projects. 


If  you  don't  have 
a  plan,  what  are 
you  monitoring 
against? 
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What  does  this  mean?4 


OPF  SP  3.4  Incorporate  Process-Related 
Experiences  into  the  Organizational  Process  Assets 

□  Incorporate  process-related  work  products,  measures, 
and  improvement  information  derived  from  planning  and 
performing  the  process  into  the  organizational  process 
assets. 


This  includes  capturing  experiences  related  to 

deploying  processes! 
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Process  Deployment  and  Monitoring 


Who  is  responsible  for  process  adoption? 


Process  deployment  and  monitoring  is  an 
organizational  activity: 

□  Projects  are  responsible  for  using  current 
processes,  but  the  process  focus  group  is 
responsible  for  making  sure  projects  can 
succeed  in  this  endeavor 

□  Monitoring  of  process  deployment  needs  to 
occur  at  an  organizational  level,  projects  have 
lots  of  other  things  to  worry  about 
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Process  Deployment  and  Monitoring 


Common  Deployment  Pitfalls 


□  Assuming  training  ensures  adoption 

□  Interacting  only  with  “process-friendly”  people  when  deploying 

□  Discouraging  feedback  during  deployment 

□  Assuming  that  no  feedback  =  adoption 

□  Mandating  immediate  adoption  of  process  changes  (without 
consideration  to  rework  or  other  project  impacts) 

□  Assuming  when  people  don’t  perform  the  process,  they  are 
wrong  and  the  process  definition  is  right 

□  Assuming  a  deployment  plan  is  not  necessary 

So  how  do  I  avoid  the  pitfalls? 
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Avoiding  the  Pitfalls 


□  Evaluate  which  projects  should  adopt  the  change  and 
when 

□  Evaluate  how  difficult  deploying  the  change  will  be 

□  Develop  a  strategic  deployment  plan  conveying  in  a  few 
bullets  how  the  deployment  will  be  accomplished 

□  Develop  a  detailed  tactical  plan  for  ensuring  all  projects 
that  should  adopt  the  change  actually  do 

□  Monitor  the  deployment  until  adoption  is  achieved 
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Process  Deployment  and  Monitoring 


Which  projects  should  adopt  the  change  and  when? 


There  are  several  considerations: 

□  Minimizing  impact  to  active  projects 

□  Urgency  of  adoption  of  this  process  change 

□  Minimizing  length  of  process  deployment  and  monitoring 

□  Minimizing  different  process  versions  in  use  at  one  time 


There  are  also  several  options: 

□  All  projects  adopt  immediately 

□  All  new  projects  adopt  at  start-up 

□  All  projects  that  have  not  yet  reached  a  certain  life  cycle  phase 

□  All  projects  for  a  particular  customer  or  product 

□  All  projects  using  a  certain  life  cycle 
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Process  Adoption  Considerations-  Example  #1 


Example:  New  Design  Document  Template  to  be 

deployed 


Certainly  all  new  projects  should  adopt  current 
processes,  but  what  about  active  projects? 
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Process  Adoption  Considerations-  Example  #2 


Example:  Status  Report  Format  Change  to  be  deployed 


Certainly  all  new  projects  should  adopt  current 
processes,  but  what  about  active  projects? 
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Process  Deployment  and  Monitoring 


How  hard  will  it  be  to  deploy  this  change? 


There  are  several  considerations: 

□  How  much  impact  will  the  change  have? 

□  Do  we  anticipate  resistance  for  some  reason? 

□  What  is  the  consequence  of  the  change  not  being  adopted? 


There  are  also  several  options: 

□  PPQA  auditing  (process  or  work  product)  may  be  sufficient  (all 
projects  or  sampled) 

□  Class  room  training  (with  or  without  exercises) 

□  One-on-one  mentoring 

□  Feedback  collection  at  point  of  implementation 

□  Pilot  before  deploying 
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Process  Deployment  and  Monitoring 


Strategic  Plan 


Before  getting  into  the  details,  develop  a  strategy,  for  example: 


□  Our  goal  is  for  all  new  projects  starting  after  October  1  to  adopt 

the  new  process 

a ) 

□  We  plan  to  retire  the  current  process  by  May  of  next  year 

Q. 

□  Projects  will  receive  training  on  the  new  process  at  project 

£ 

start-up  -  annual  refresher  training  will  include  an  overview  of 

(0 

the  new  process 

X 

LU 

□  We  will  monitor  adoption  by: 

□  Conducting  lessons  learned  specific  to  this  process 

□  Auditing  all  projects  for  the  first  year 
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Process  Deployment  and  Monitoring 


Tactical  Plan  -  Overview 


Process  deployment  also  requires  a  tactical  plan: 


□  Develop  an  integrated  schedule  that  ties  the  process 
deployment  to  each  affected  project’s  life  cycle 


□ 

□ 

□ 


Identify  resources  needed  (facilities,  people,  materials) 
Schedule  training,  PPQA  audits,  other  deployment  support 

Schedule  monitoring  (evaluation  of  results)  >i 

□  Review  PPQA  results  \  h}  | 

□  Review  training  feedback 

□  Review  lessons  learned 


For  some  changes,  we'll  need  a  way  to  determine 
whether  to  stop  and  re-start  the  deployment 
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Tactical  Plan  -  Measures 


Establish  measures  for  determining  when  you’re  done: 

□  When  do  we  expect  all  projects  to  be  using  the  new  process? 

□  How  long  do  we  have  to  keep  monitoring? 

□  Monitor  first  “n”  uses  of  the  process? 

□  Monitor  through  a  certain  date? 

□  How  will  we  determine  success  of  the  deployment? 

□  Satisfactory  PPQA  results? 

□  %  training  attendance  by  teams? 

□  Minimum  survey  results  on  deployment? 

□  Feedback  on  deployed  process  (#  of  change  requests  or 
tailoring  requests)? 
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Deploying  processes  at  project  startup 


Project  start-up  should  include  providing  guidance  on  defining 
and  implementing  the  project’s  defined  process: 

□  Provide  a  reminder  of  recent  process  changes 

□  Check  that  everyone  is  trained  on  recent  process 
changes 

□  Check  for  new  team  members  and  people  who  have 
changed  roles 

□  If  this  will  be  one  of  the  monitored  projects,  remind  the 
team  of  this 


Process  deployment  needs  to  consider  impacts  on  the 

project  start-up  process 
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Process  Deployment  and  Monitoring 


Ready,  Set,  Go! 


Make  sure  you  are  ready  to  deploy: 

□  Assets  are  in  the  PAL 

□  Tailoring  criteria  is  defined 

□  Communicate: 

□  Who  must  comply  and  by  when 

□  What  training  will  be  provided  and 

□  What  additional  assistance  will  be 

□  Why  the  change  is  being  made 

□  Training  scheduled 

□  PPQA  audits  scheduled 

□  Support  resources  secured 


when 
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What  will  monitoring  tell  us? 


Training 

Results 


Was  attendance  adequate? 

Were  training  materials  adequate? 
Additional  or  different  training  needed? 


PPQA  Audit 
Results 


Do  audit  results  reflect  adoption? 

Is  audit  efficient  or  overly  time  consuming? 


Lessons 

Learned 


Are  changes  to  the  process  needed? 

Have  future  improvements  been  identified? 


Validation 


Does  process  change  achieve  business 
objective? 


(D  rv. 

E  ^ 


c/) 

(/) 

CD 

O 

O 
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Process  Deployment  and  Monitoring 


How  will  I  know  the  deployment  worked? 


Ultimately,  successful  adoption  means: 

□  PPQA  audits  indicate  the  process  is  being  followed 

□  Lessons  Learned  indicate  use  of  the  process  and 
identification  of  potential  improvements  rather  than 
problems 

□  Evidence  of  process  use  (measures,  work  products, 
tailoring  requests)  are  being  collected 

□  Feedback  on  the  defined  processes  and  assets  is 
used  to  improve  them 
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What  happens  once  the  process  is  adopted? 


□  Training  on  the  new  process  remains  part  of  overall 
training 

□  PPQA  audits  continue  to  be  performed  (but  possibly 
on  smaller  sampling  of  projects) 

□  Process  improvements  to  training  materials,  PPQA 
assets,  deployed  assets,  and  deployment  processes 
are  captured 

□  Adoption  measures  are  captured 
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Process  Deployment  and  Monitoring 


Evaluating  the  deployment  process 


Evaluate  and  measure  the  deployment  process: 

□  Surveys  or  lessons  learned  sessions  with  teams 
involved  in  the  deployment 

□  Elapsed  time  to  adoption  versus  size  of  change 

□  %  of  projects  that  used  support 

□  #  of  deployment  re-starts 


25 


©2010  Natural  SPI 


Process  Deployment  and  Monitoring 


Summary 


Deployment  planning  can  mean  the  difference  between  project 
deployment  success  and  failure: 

□  Planning  may  take  some  time  for  the  deployment  team,  but 
will  save  time  for  all  impacted  projects  during  deployment 

□  Projects  plan  and  manage  their  work  -  the  deployment 
team  should  do  the  same! 


Take  ownership  for  successf ul  deployment! 
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Natural  spi 


Questions 


©2010  Natural  SPI 


Process  Deployment  and  Monitoring 


Contacts  and  More  Information 


□  Susan  Byrnes,  PMP 


Natural  SPI,  Inc. 

E-mail:  susan@naturalspi.com 
□  Natural  SPI’s  web  site:  www.naturalspi.com 


Natural  spi 


SEI  Partner 
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Applying  CMMI-SVC 

to 

Process  Management 


LOCKHEED 


Allen  Eagles 
Lynn  Penn 
Dorna  Witkowski 


Agenda 


*  Process  Management  introduction 

*  Planning 

*  Implementation 

*  Results 

*  Plans  ahead 
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Process  Management 


*  Reports  to  the  CTO/Engineering  &  Technology 

*  Responsible  for: 

-  Standards  and  model  assessment,  internal  and 
external 

•  CMMI,  ISO,  AS9100 

•  Tools  to  support  assessments:  Common  Audit 
Database,  Program  Profile  Database,  etc. 

-  Best  Practice  Libraries,  R&D  process  support, 
process  collaboration  with  Product  Lines,  Policy 
board  chair  and  facilitation 

-  Performance  Analytics  tools  and  metrics 

-  Lean/Six  Sigma  Operating  Excellence 

*  Reorganized  in  2010  to  be  more  service  and  project 
focused 
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Company 


Product  Lines 


n i  i 

PL1  PL2  PL3  PL4 


Executive 

Vice  President 

Chief  Te 
Off 

chnology 

icer 

IRAD  Tech  Cyber  Engineering 
Centers  Security 


Functions:  HR, 
CIO,  Finance,  etc. 


Process 

Management 


X 


Process  Management  Organization 

July  26,  2010 


Staff 

■  Dottie  Acton* 

■  Mary  Busby 

■  Jennifer  Curtis 

■  Allen  Eagles 

■  Gary  English 

■  Paul  Gehman 

■  Barbara  Geshwind 

■  Marc  Gewertz 


Greg  Kerchner 
Len  Mikolajczak 
Nicole  Tucker* 

Dick  Warren 
Suzanne  Weisenberger 
Joan  Weszka 
Bob  Yates 


IS&GS  Engineering  &  Technology 


Technical  Support 


Process  Management 

Service  Initiatives 


x 


•  CMMI: 

-  Provide  training,  consulting,  SCAMPI  B/C  preparation  and 
execution,  SCAMPI  A  preparation  and  support 

•  ISO  and  Management  Systems: 

-  Provide  training,  consulting,  internal  auditing,  process 
monitoring,  supervision  of  ISO  and  AS9100  audits 

•  Performance  Analytics 

-  Establish  baselines;  perform  benchmarking,  monitoring, 
control,  cost  and  schedule  modeling,  affordability  and 
productivity,  basis  of  estimates,  predictive  analytics 

•  Operating  Excellence  -  Lean/Six  Sigma: 

-  Provide  Lean/Six  Sigma  training,  event  facilitation,  process 
improvement  reporting,  Product  Line  process 
improvement  facilitation 


In  2010  each  initiative  was  required  to  create 
a  project  plan,  a  schedule,  and  a  service  catalog 


Why  use  CMMI-SVC 


X 


•  To  provide  better  service  to  our  internal 
customers 

•  To  become  more  disciplined 

•  To  establish  continuity 

•  To  conform  to  what  we  expect  from  our 
programs 

•  To  expand  knowledge  of  problems  and  issues 
we  might  face  from  practitioners 


Quick  Overview  of  CMMI-SVC 
Specific  Process  Areas 

•  Service  Delivery  (SD)  -  Deliver  services  in  accordance  with  service  agreements 

•  Capacity  and  Availability  Management  (CAM)  -  Ensure  effective  service  system 
performance  and  ensure  that  resources  are  provided  and  used  effectively  to  support 
service  requirements 

•  Incident  Resolution  and  Prevention  (IRP)  -  Ensure  timely  and  effective  resolution  of 
service  incidents  and  prevention  of  service  incidents  as  appropriate 

•  Service  System  Transition  (SST)  -  Deploy  new  or  significantly  changed  service  system 
components  while  managing  their  effect  on  ongoing  service  delivery 

•  Service  Continuity  (SCON)  -  Establish  and  Maintain  plans  to  ensure  continuity  of 
services  during  and  following  any  significant  disruption  of  normal  operations 

•  Service  System  Development  (SSD)  -  Analyze,  design,  develop,  integrate,  verify  and 
validate  service  systems,  including  service  system  components,  to  satisfy  existing  or 
anticipated  service  agreements 

•  Strategic  Service  Management  (STSM)  -  Establish  and  maintain  standard  services  in 
concert  with  strategic  needs  and  plans 
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Planning  the  CMMI-SVC  Gap 
Analysis 


•  Scope: 

-  Within  Process  Management  -  only  our  4  major 
initiatives  and  management 

•  Did  not  cover  tools  and  other  services 

-  Within  the  model  -  only  looked  at  the  specific 
practices  of  the  Service  PAs 

•  Including  the  addition,  Service  System 
Development 
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Planning  the  CMMI-SVC  Gap 
Analysis 


•  Sequence  of  Process  Areas  to  review: 

-  Phase  1:  Service  Delivery,  Incident  Resolution  (and 
Prevention),  Service  Continuity 

•  We  focused  on  process  areas  where  evidence  was 
available 

-  Phase  2:  Service  System  Development,  Strategic 
Service  Management 

•  As  the  year  progressed,  there  was  time  to  show  evidence  of 
strategy  and  development  of  new  or  modified  services 

-  Phase  3:  Capacity  and  Availability  Management, 
Incident  (Resolution  and)  Prevention,  Service 
System  Transition 

•  Most  challenging  for  a  small  service  group 
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Planning  the  CMMI-SVC  Gap 
Analysis 

*  Method  -  work  product  focus 

-  Study  the  CMMI-SVC  Process  Area 

-  Consider  Specific  Practice  Work  Products  in  the 
model 

-  Look  at  Process  Management  Work  Products 
•  That  will  meet  the  intent 

-  Interview  Project  Leads  to  identify  gaps 

-  Map  the  collected  data  to  the  Specific  Practices 

-  Publish  the  gap  analysis,  assign  actions  and  work 
to  closure 

-  Track  completion  of  action  plans 
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Implementation  of  the  Plan 

Sample  interpretation  of  work  products  and  gap  analysis 

Service  Delivery  SP  2.1  Establish  the  Service 

Delivery  Approach 

Model  has  example  work  products: 

•  Service  delivery  approach 

•  Contact  and  roster  lists 

•  Service  request  criteria 

•  Internal  and  external  status  reporting 

Initially,  looked  for  Process  Management  work 
products : 

•  Documented  service  approach 

•  Contact  and  team  member  lists 

•  Customer  and  senior  management  status  reports 
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Implementation  of  the  Plan 

Sample  interpretation  (continued) 

Actual  work  products  found: 

•  Service  delivery  approach  provided  in  project  plan 

Service  Delivery  sample  from  Project  Plan:  Product  Lines,  Programs,  or  Senior 
Management  may  request  the  services  identified  in  the  Service  Catalog.  The  CMMI 
project  will  provide  staffing  for  the  requested  service  and  establish  an  agreement  with 
the  requester  with  respect  to  funding,  timeframe  of  the  service,  work  products  to  be 
provided  with  the  service,  and  expectation  of  the  results  of  the  service. 

•  Contact  list;  Team  member  list 

•  Internal  status  reports 
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Implementation: 


•  Along  the  way,  we  met  some  resistance... 


-  Critical  milestones  such  as  ISO  Audits,  Executive 
Directives,  and  Corporate  Milestones  interfered  with 
implementation 

-  “Don’t  bother  me  I’m  too  busy  with  this  ISO  Audit.” 


-  Limited  staff  delayed  resolution  of  actions 

-  “I  don’t  have  time  to  do  this,  one  of  my  2  staff  members  was 
reassigned  and  I’m  behind  in  my  schedule.” 

-  “With  only  2  and  V2  people  I  don’t  think  my  project  meets  the 
definition  of  a  service  as  referenced  in  the  model!” 


■  Perceptions 

-  “I  don’t  need  to  test  continuity,  I  have  everything  on  our 
web  site.” 


Now  we’re  prepared  for  programs!! 
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Implementation: 

...and  acceptance 


*  Showed  project  leaders  why  certain  activities 
identified  in  the  service-specific  process  areas  can 
actually  benefit  them,  their  project,  and  their 
customers 


-  Examples: 

•  Tested  continuity  plans,  with  reported  results 

•  Incidents  tracked  and  workarounds  determined; 
future  analysis  planned 

•  Guidebook  on  developing  /  revising  a  service 
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Results 


*  Interpretation  of  the  model 

-  Implementing  the  model  didn’t  require  fancy  tools, 
spreadsheets,  etc. 

-  We  held  a  brainstorming  session  to  define 
“Incidents”  for  our  service  initiatives 

*  Methodology 

-  We  adjusted  our  methodology  along  the  way 

•  Changed  planned  sequence  of  PAs  after  phase  1 

•  In  later  phases,  spent  more  time  on  pre-interview 
discovery  to  look  for  evidence  already  available 

•  Adjusted  methods  when  we  found  true  systemic 
gaps  for  SSD  and  IRP 
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Results 


*  Gap  Analysis  -  systemic  problems 

-  No  documented  approach  to  developing  a  new 
service 

-  No  documented  continuity  plans 

-  Incidents  not  defined 

-  Incident  tracking  not  defined 

-  No  formal  service  level  agreements  to  set 
expectations 
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Results 


*  Developed  and  published  service  catalogs  for  each 
project  area  and  the  other  services  we  offer 


Service  Catalogue/ 


catalogue 

Point  of  Contact 

Related  Links 

CMMI 

Dorna 

V/itcowski 

610-354-2422 

SCAMPI  A  Trsinsd  ETiDlovees 

CMMI  FAOs 

Available  Too  s  for  CMMI 

Reviews 

Links  :o  Certifications  and  ADSs 

ISO  &  Manaaement  Systems 

viike  La  Barge 

Obta  nnq  Copies  of  Standards 

1  310-354-5077 

SAE  AS9100C  FAOS 

PerforniancelAnalYtics 

Pe:e  McLoone 

610-354-6936 

LN21  3peratina  Excellence 

Ker  Csdds 

856-485-5115 

Other  Services 

Sea  Cataogue 

I5&G5  Process  Management:  Other  Initiatives 

[Tools  and  Services] 


Carcunon  Audit  Database 


husinnys  Pror*^  ^mur  Fin-'in  i  jtiPRUf 

EEGL^^MI 

tiecpnh  M  aiwement 


IS&GS  Process  Management:  ISO  Related  Services 

[ISO  9001:  SAE  AS9100;  ISO  20000;  ISO  27001| 

Training: 

Internal  Audits: 

Certifications: 

Soecification  Training: 

Mtjuuj^  T  t  ,rir  x 

Plaimni;  Schadul<tg;  Ixicttsag 

|1.5fwto  4h»| 

*  ISC  9001  •  i  IQ  2(4  Q0 

Internal  Program 

Audits 

initial  Certifications 

‘  SAf<S91CQ  •  ISOI70Q1 

Surveillance  Audits 

*  Editor  ruieing 

Internal  Functional 

Audits 

Re-Certllkatiom 

Custom  Training:  rousing 

Facilitation  iwith 

Ndt^len  jw/J«ternal 

Aktftlto#*) 

•  IS46SC«rvll«eog» 

Eslcraa!  Auditors] 

Cyber  Security:  Audits  & 

Process  Monitoring 

M-U.WI  U.V, 

Guiconce:  (CALiPER) 

Coordination 

IUU  tNUllCUIfiM  | 

Services  and  Products 


•  Engineering /Services  Analytics  SoE 
- 8 asel in es a n d  B e n ch m arking 
-Monitoring and  Control 
-Cost  and  Schedule  Modeling/ 

Basis  of  Estimates 
-  Predictive  Analytics 


I5&GS  Process  Management:  CMMI®  Related  Services 

[CMNII-DEV*,  CMMI-SVC*] 


Training: 

SEI  Mode)  Training: 

■  IptroJualaniaCM-W-iYC  ' 
*  WMhSVC  ’  SL'ppItmtfti  |L 

CtBtom  Training: 

■  SCAMrt*“  A  r 
TnMInf 

'  CMM'  Vl.3  Uu  I  r  »D«I- 
•  Ovnrvlaw  c4  Kijgh  Maturity 


Consulting: 

Proposals 

SCAM  FI'  A  Preparation 

FMDP  Modules 

SEI  Protects 

Froduet  Line  and 
ProRrarvi  Const!  It  Irm 


Appraisals: 

SCAMPI  /  Class  C 

SCAMPI/ Class  B 

Continuous  Appraisal 

Method  (CAM  1 

Custom  Appraisals 

Gene  rat  Contractor  for 

SCAMPI  A 


IS&GS  Process  Management 
LM21  Operating  Excellence  Service  Catalogue 


Coramunlc-alJony 


i  r>tr«afr  uctu  nr  f 


BlbcLBah  CrriJtlMEli^t  SIAt 

-  Slocl  Bel!  Vediw  Sirwra 

(k'Hcn  Bolt  -  &f  c-ofi  e-iHc  VSM 

l<iiiL>«1ii  SIA  Mentor  I  ns:  |,,'K 

-  RIjw  k  Bp  IE  '  PDK 

Apollo  RtA 


Pro(r*m  Eyceflmee 


i  wn  ormr  m 


QELA.H  ^l.'iL»r[prly 
Ihj**ng94  Ares 
Hewewl 


Prote+i  MOR  OE  £*cii|l*nc*  PJcn 


ItataL.!  CP  UbE 


Overview -si 
L-V1  IE  I  Ob 
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Results 


Established  incident  and  request  logs 


3 

A 

Date  ▼  Request  ▼ 

C  |  D  |  E  i  F  |  G  |  H  | 

Requestor  -  Type  ▼  ▼  Status  ▼  Respon  Time 

18 

19 

6/2/2010  Request  for  updated  CMMI  Proposal  writeup 

Chris  Conway  Research  C  5  min 

20 

6/2/2010  Request  to  follow  up  on  CMMI  ratings  in  UK 

Lynn  Penn  Research  .25  hour  C  15  min 

21 

6/2/2010  question  on  site  CMMI  rating 

Rhonda  Hammond  Quick  Email  C  6  min 

22 

6/3/2010  Phone  call  asking  about  ITIL  class 

Jessy  Calonje  Research  .25  hour  C  15  min 

23 

6/3/2010  Request  for  copy  of  IS&GS-Defense  ADS 

Lynn  Penn  Quick  Email  C  5  min 

24 

6/3/2010  Charge  for  CMMI  class 

Patrick  Diskin  Quick  Email  C  10  min 

25 

6/6/2010  FDOC  request  to  review  OT  process  assets 

Bernice  Mays  Consulting  1  hour  C  60  min 

26 

6/17/2010  Request  for  posting  SEI  analysis  of  ROI 

Lynn  Penn  Research  .25  hour  C  15  min 

27 

6/17/2010  Request  to  withdraw  SEI  analysis  of  ROI 

Joan  Weszka  Quick  Email  C  5  min 

28 

6/17/2010  FDOC  request  to  review  PP/PMC/IPM  assets 

Bernice  Mays  Consulting  1.5  hours  C  90  min 

29 

6/22/2010  Request  for  information  on  VoC 

30 

31 

32 

33 

34 

35 

36 

37 

38 

6/29/2010  Question  about  Denver  CMMI  class 

7/7/2010  Request  to  review  FDOC  CM  process  assets 
7/9/2010  Request  for  lessons  learned  or  strategies  on  V  1 
7/12/2010  Request  to  review  FDOC  MA  process  assets 

Info  on  our  process  for  tracking  authorization  2 

ISO  Related 

Ad  Hoc 

activites 

Date  of 

Request  Task  Description  Who's  working  it:  Approx  effort: 

7/16/2010  status 

ISO  Certification  queries 

GME-MS 

1 

7/16/2010  Question  about  being  able  to  end  a  class  by  5I-^ J 
Question  about  whether  an  initiative  was  a  gc- 
7/19/2010  example  for  OID 

7/19/2010  Request  for  CMMI  charts  for  Affordability  task  36 

ISO  Sponsorship  for  LM  Support  on  Hubble  Space  Telescope 

GME  -  MS 

1 

AS9100  REV  C  LEAD  AUDITOR  TRAINING  planning  for  Bruce  and  Gary 

GME-MS 

2 

OASIS  S:N  ratio 

GME-MS 

3 

39 

7/19/2010  Request  for  Services  Strategy  charts  37 

thinking  about  CAPs  for  CARs 

GME-MS 

2 

40 

7/20/2010  Request  to  resend  CMMI  cost  data  38 

Security  measurement  input 

GME-MS 

1 

41 

42 

7/22/2010  Request  ot  review  Engineering  process  gg 

7/22/2010  Request  to  review  audit  processes 

SOE-0023a  and  SOE-0090  support 

GME-MS 

2 

Marcus  Mclnnis  strategy  writeup  support 

GME-MS 

2 

•  <  ►  w  Requests  Incidents  Customer  Complaints  f'  ^ 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

QE  Steering  Panel  Working  Session 

GME-MS 

0.5 

CCSI  -  Cyber  Security  Team  Monthly  Review 

GME  -  CS 

3 

27001  for  aero 

GME  -  CS 

2 

27001  for  space 

GME  -  CS 

1 

Follow  up  on  CD  from  John  Thompson 

GME  -  CS 

2 

ISO  27001  and  2010  Program  list 

GME  -  CS 

3 

LM  VTIS  -  CIPP-IT  Certification  on  Privacy  -  877-771-7176  pc  163436 

GME  -  CS 

1 

~  monthly  update  to  action  item  /  to  do  list  for  Lynn 

GME  -  stuff 

2 

checking  on  blackberry  version  and  exploit  non-issues 

GME  -  stuff 

0.5 

ICE  Presentation  (Immigration,  Customs  and  Export)  by  Special  Agent 

GME  -  stuff 

0.8 

email  beat  down 

GME  -  stuff 

20 

LM  Symantec  Webinar  Today 

GME  -  stuff 

1 

San  Diego  Wildfire  support  for  BC/DR 

GME  -  stuff 

1 

Services  Strategy  and  Repository  meeting 

GME  -  stuff 

0.5 

Standard  Meeting  for  Company  Points  of  Contact 

GME  -  stuff 

1 

Memorial  dav 

GME  -  stuff 

9 
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Results 


*  Established  continuity  plans  and  tested  the  plan 


Service  Continuity  Plan  for  CMMI  Project 

Service  Continuity  Test  -  March  2010 

TheessentialfunctionsoftheCMMI  Project  include  performing  as  team  m 

as  planning  and  performingSCAMPI/ClassB'sandC's.  These  are  the  functi 

Test  8 1 :  CMMI  Project  Lead  Backup 

customer's  schedules  and  may  impact  their  business  goals. 

The  essential  resources  of  the  CMMI  Project  are  the  staff  performingthe  e 

Test  conducted  3/7/10  -  3/11/10;  black  text  is  the  test  from  the  Service  Continuity  Plan;  Aqua  testis  the 

tools  they  use  in  performingthese  functions. 

result  of  the  test. 

All  pertinentfiles  and  information  will  be  stored  on  the  Process  Manager™ 

Preparation : 

folder 

•  Exceptions  are  Risk  Managementfiles  and  MOR  files  which  are  stoi 

•  Identify  2  customers  to  participate  in  test 

Management  Unity  site  in  other  standard  folders. 

o  Marcia  Miller  and  Lloyd  Brown  from  ISStGS- Defense  supported  the  test 

•  This  should  include  all  work-in-progress  files  for  planned  work 

•  Identify  1  internal  CMMI  staff  member  to  participate  in  test 

Unity  sites  are  backed  up  on  a  daily  basis  by  Enterprise  Business  Sy 

o  Barbara  Geshwind  supported  the  test 

The  CMMI  Project  Lead's  back-up  is  the  Director,  Process  Management. 

•  Tell  Director,  Process  Management  of  testto  be  done 

•  On  a  periodic  basis,  the  CMMI  Project  Lead  will  ensure  the  Directo 

o  Director,  Process  Managementwas  told  the  week  before  the  testthat  it  would  be  done 

knowsofall  current  activity.  This  can  be  done  at  weeklyface-to-f< 

•  CMMI  Project  Lead  sets  Out  of  Office  message  to  have  customers  contact  Director,  Process 

Monthly  Operating  Reviews,  and  through  emails. 

M  a  n  age  m  e  nt  f  o  r  CM  M I  q  u  e  stio  ns  o  r  a  ctivitie  s 

The  back-upforCMMI  Project  staff  doing  planned  projects  forthe  CMMI  P 

o  Out  of  office  message  set  on  Email  and  Voice  Mall 

Lead. 

•  Contact  customers  and  requestthatthey  email  or  call  with  questions  on  CMMI 

•  On  a  periodic  basis  (nominally  weekly),  CMMI  Project  staff  will  ens 

o  Customers  contacted 

Lead  knowsthe  status  of  all  activity,  as  well  as  points  of  contact  foi 

•  Contact  internal  CMMI  Staff  member  and  request  that  he/she  email  or  call  with  question  on 

tasks 

□  Infernal  staff  member  contacted 
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Results 


*  Established  service  agreements  and  received 
feedback  from  customers 


CMMI  Project  Service  Agreement 

Requestor:  Click  here  to  enter  text. 

Description  of  Activity: 

Click  here  to  enter  text 
Timeframe:  Click  here  to  enter  text. 

Work  Products  to  be  provided: 

Click  here  to  enter  text 

Requestor's  responsibilities: 

Click  here  to  enter  text 

Expected  Results: 

Click  here  to  enter  text 

Funding:  Click  here  to  enter  text. 

CMMI  Project  Staff:  Click  here  to  enter  text. 

Date  Approved  by  Requestor  and  CMMI  Project: 

J 


CMMI  Project  Service  Agreement  -  Feedback  on  Results 
(To  be  completed  after  service  provided) 

Please  send  this  information  to  your  Process  Management  Product  Line  Point  of  Contact,  the  CMMI  Project  Manager,  or 
the  Director  of  Process  Management: 

•  Process  Management  Product  Line  Points  of  Contact:  Civil  iDaul.r.gehman@lmco.com).  Defense 


•  CMMI  Project  Manager:  dorna.witkowski@ lmco.com 

•  Director,  Process  Management:  marv.lvnn.Denn@lmco.com 

Response  provided  by:  Click  here  to  enter  text. 

Please  indicate  your  satisfaction  with  the  provided  service  by  selecting  a  value  1  through  5  where  1  is  not 
satisfied  and  5  is  completely  satisfied.  Please  feel  free  to  elaborate  by  entering  comments.  To  complete 
the  section  below ,  "unprotect"  the  document  by  going  to  the  "Developer"  tab,  Protect  Document,  and 
remove  the  check  before  "Restrict  Fojjnatting^and  Editing"  OR  request  an  unrestricted  copy  from 
dorna.  witkowski@lmco.  com. 

Satisfaction  with  overall  service  provided: 

1  2  3  4  5  Comments: 

Satisfaction  with  the  agreed-upon  work  products: 

1  2  3  4  5  Comments: 

Satisfaction  with  the  agreed-upon  results: 

1  2  3  4  5  Comments: 

Satisfaction  with  meetingthe  timeframe: 

1  2  3  4  5  Comments: 
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Results 


•  Also: 

-  Wrote  guidance  for  developing  services 

-  Working  on  a  service  strategy  for  our  entire 
business  area 

•  A  service  sample  repository 

•  Maps  of  service  components  to  models  and 
standards 
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What  was  easy  to  implement 


•  SD  -  Service  Delivery 

-  Providing  our  services  in  a  more 
professional  and  organized  manner 

•  The  IR  of  IRP  -  Incident  Resolution 

-  Identifying  and  resolving  incidents 

•  SCON  -  Service  Continuity 

-  Establishing  and  testing  continuity  plans 

•  SST  -  Service  System  Transition 

-  Providing  a  new  service  in  an  orderly, 
controlled  manner 
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What  was  easy  to  implement  (cont.y 


x 


•  STSM  -  Strategic  Service  Management 

-  Establishing  plans,  service  catalogs  and 
levels  of  service 

•  SSD  -  Service  System  Development 

-Although  we  didn’t  have  anything 
documented  initially,  following  a  standard 
method  of  developing  new  or  modified 
services  will  provide  a  more  disciplined  and 
thorough  method  that  we  can  easily  follow 
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What  is  hard  to  implement 


*  CAM  -  Capacity  and  Availability  Management 

-  Although  we  know  and  can  manage  our  capacity 
and  availability,  a  “Service  System  Representation” 
is  very  difficult  for  a  small  people-related 
organization 

•  The  P  of  IRP  -  Incident  Prevention 

-  Because  incidents  are  rare  and  there  is  usually  no 
commonality  between  them,  it  is  difficult  to 
establish  prevention  mechanisms 
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Plans 


*  Expand  to  Core  Process  Areas  in  2011 

*  No  plans  for  a  SCAMPI  B/C  at  this  time  as  we 
are  still  working  to  close  gaps 

*  Use  the  experience  gained  for  future  adoption 
across  the  business  area 
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Questions 
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Improving  Operational  Resilience 
Processes 

CERT  Resilience  Management 
Model  (RMM) 

10th  Annual  CMMI  Technology 
Conference  and  User  Group 


Rick  Barbour  &  Rich  Caralli 
CERT 

Software  Engineering  Institute 


Software  Engineering  Institute 


Carnegie  Mellon 


©2010  Carnegie  Mellon  University 

Sponsored  by  the 

U.S.  Department  of  Defense 


Agenda 


>  What  is  CERT®-RMM? 

>  Risk,  Resilience  &  Convergence 

>  Overlap  between  CERT-RMM  &  CMMI  process  areas 

>  CERT-RMM  as  an  organizing  principle 

>  CERT-RMM  Current  Activities 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 

©2010  Carnegie  Mellon  University 
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What  is  CERT®  RMM? 


The  CERT®  Resilience 

Management  Model  (CERT- 
RMM)  is  a  capability  model  for 
managing  and  improving 
operational  resilience. 


Positions  operational  resilience 

in  a  process  improvement  view 

Includes  26  “process  areas” 

Focuses  on  the  operations  phase 


of  the  lifecycle 

•  Defines  “maturity”  through 
“capability  levels”  consistent  with 
CMMI 

•  Uses  CMMI  architecture  for  ease 
of  adoption 

•  Includes  a  “continuous 
representation”  for  agile  adoption 


Distinguishing  features  of  CERT®'RMM 


CERT-RMM  brings  several 
innovative  and  advantageous 
concepts  to  the  management 
of  operational  resilience. 


•  The  convergence  advantage: 

merging  the  disciplines  of 
security,  BC/DR,  and  IT  ops  into 
a  single  model 


•  The  process  advantage: 

elevating  these  disciplines  to  a 
process  view,  useful  as  an 
integration  and  measurement 
framework 

•  The  maturity  advantage: 

provides  a  foundation  for  practical 
institutionalization  of  practices — 
critical  for  retaining  these 
practices  under  times  of  stress 


CERT-RMM  background 


CERT-RMM  began  as  research  •  Literary  review  and  affinity  analysis  of 
into  the  application  of  process  over  800  standard  practices  security, 

improvement  and  maturity  BC/DR,  and  IT  ops  communities 

model  approaches  to  security  .  Examination  of  body  of  knowledge  of 
management.  high-maturity  organizations 

•  Codification  of  model  using  trusted 
CMMI  architecture  and  concepts 

•  Benchmarking  and  piloting  in  the 
banking/finance  community,  defense 
contractors,  and  US  government 
federal  civilian  agencies 


^==-  Software  Engineering  Institute 


Carnegie  Mellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 

©2010  Carnegie  Mellon  University 
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CERT-RMM  at  a  glance 


Resilience,  Convergence  &  Risk 


^=-  Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 

©2010  Carnegie  Mellon  University 


Operational  resilience 


Resilience:  The  physical  property  of  a 
material  when  it  can  return  to  its 
original  shape  or  position  after 
deformation  that  does  not  exceed  its 
elastic  limit  [wordnet.princeton.edu] 


Operational  resilience:  The 
emergent  property  of  an 
organization  exhibited  when  it 
continues  to  carry  out  its 
mission  after  disruption  that 
does  not  push  it  beyond  its 
operational  limit 


egie  Mellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Convergence 


A  fundamental  concept  in  managing  operational  resilience 

Refers  to  the  harmonization  of  operational  risk  management  activities 
that  have  similar  objectives  and  outcomes 

Operational  risk  management  activities  include 

•  Security  planning  and  management 

•  Business  continuity  and  disaster  recovery 

•  IT  operations  and  service  delivery  management 

Other  support  activities  may  also  be  involved — communications, 
financial  management,  etc. 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Operational  resilience  &  operational  risk 


Security  and  business  continuity  are  not  end-states;  they  are 
continuous  processes 

Effective  operational  risk  management  requires  harmonization: 
convergence  of  these  activities  working  toward  the  same  goals 

Operational  resilience  emerges  from  effective  operational  risk 
management 


Actions  of 
people 


Systems  & 
technology 
failures 


Failed  internal 
processes 


External  events 


Operational  resilience  and  convergence 


Convergence  directly  affects  the  level  of  operational  resilience. 

Level  of  operational  resilience  affects  the  ability  to  meet  organizational 
mission. 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 

©2010  Carnegie  Mellon  University 


11 


Protection,  sustainability,  and  risk 


Basic  risk  equation  Protection  &  sustainability 


consequent 


condition 


RISK 


Operational  resilience  requires  optimizing  these 
strategies  in  a  way  that  minimizes  operational  risk  (to 
the  associated  services)  and  is  resource  efficient:  the 

management  challenge  of  operational  resilience. 


Information 


protect  | 


Manage  Conditions  of  Risk 


Manage  Consequences  of  Risk  1 

' . T . 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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A  service  view 


Organization 

Mission 


A 


Service 

Mission 


The  organization  meets  its  mission  when  high-value  services 
in  the  organization  meet  their  missions. 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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The  object  of  improvement 


CERT-RMM  position  in  lifecycle 


Plan 


Design 


Acquire 


Develop  Deploy 


l  l 

I  U 

i _ ! 


t 


CM  Ml- A  CQ 


CMMI-SVC 


DEVELOPMENT 


^Operate 


Decommission 


CERT-RMM 


OPERATION 


Software  Engineering  Institute  CarnegleMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Overlap  between  CERT-RMM  &  CMMI  process 
areas-1 


CMMI  Models  Process  Areas 

Equivalent  CERT-RMM  Process  Areas 

CAM  -  Capacity  and  Availability  Management 

(CMMI-SVC  only) 

TM  -  Technology  Management  Availability  management  is  a 
central  theme  of  CERT-RMM  this  includes  PAs:  RRD,  RRM,  EC,  KIM, 

PM,  TM 

IRP  -  Incident  Resolution  and  Prevention 
(CMMI-SVC  only) 

IMC  -  Incident  Management  and  Control 

MA  -  Measurement  and  Analysis 

MA  -  Measurement  and  Analysis  is  carried  over  intact 
from  CMMI. 

OPD  -  Organizational  Process  Definition 

OPD  -  Organizational  Process  Definition  is  carried  over 
from  CMMI,  but  development  life-cycle-related  activities  and 
examples  are  deemphasized  or  eliminated. 

OPF  -  Organizational  Process  Focus 

OPF  -  Organizational  Process  Focus  is  carried  over  intact 
from  CMMI. 

OT  -  Organizational  Training 

OTA  -  Organizational  Training  and  Awareness 

OT  is  expanded  to  include  awareness  activities  in  OTA. 

REQM  -  Requirements  Management 

RRM  -  Resilience  Requirements  Management 

Basic  elements  of  REQM  are  included  in  RRM,  but  the  focus  is  on 
managing  the  resilience  requirements  for  assets  and  services, 
regardless  of  where  they  are  in  their  development  cycle. 

RD  -  Requirements  Development 

RRD  -  Resilience  Requirements  Development 

Basic  elements  of  RD  are  included  in  RRM,  but  practices  differ 
substantially. 

Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Overlap  between  CERT-RMM  &  CMMI  process 
areas-2  and  Other  Connections 


CMMI  Models  Process  Areas 

Equivalent  CERT-RMM  Process  Areas 

RSKM  -  Risk  Management 

RISK  -  Risk  Management 

Basic  elements  of  RSKM  are  reflected  in  RRM,  but  the  focus  is  on 
operational  risk  management  activities  and  the  enterprise  risk 
management  capabilities  of  the  organization. 

SAM  -  Supplier  Agreement  Management 

EXD  -  External  Dependencies  Management 

In  CERT-RMM,  SAM  is  expanded  to  address  all  external  dependencies, 
not  only  suppliers.  EXD  practices  differ  substantially. 

SCON  -  Service  Continuity 

(CMMI-SVC  only) 

SC  -  Service  Continuity 

In  CERT-RMM,  SC  is  positioned  as  an  operational  risk  management 
activity  that  addresses  what  is  required  to  sustain  assets  and  services 
balanced  with  preventive  controls  and  strategies  (as  defined  in  CTRL  - 
Controls  Management). 

TS  -  Technical  Solution 

RTSE  -  Resilient  Technical  Solution  Engineering 

RTSE  uses  TS  as  the  basis  for  conveying  the  consideration  of  resilience 
attributes  as  part  of  the  technical  solution. 

Other  Conncections: 

Generic  goals  and  practices 

The  generic  goals  and  practices  have  been  adapted  mostly  intact  from 
CMMI. 

Other  Connections: 

Continuous  representation 

CERT-RMM  adopts  the  continuous  representation  concept  from  CMMI 
intact. 

Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 
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User  Group  2010 
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Example:  CERT-RMM  as  an  organizing  principle 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Current  Approaches  to  Security  Management 


Security  by  compliance 

National  Institute  of  *'M 

Standards  and  Technology 

|  I  ■ 

1  'net 

•FISMA 

working  wiffi  mds/stry  te  foster  innovation,  trade,  security  and  fobs 

• HIPAA 
•PCI 

Security  by  adoption  of  best 
practices 

•ISO  1 7799 
•DISA  STIGs 
•  Vendor  guides 


Protect 
Patient 
Information 


Q 


I' 


Result: 

Uneven  use  of  limited  resources 


Software  Engineering  Institute  CarnegieMellon 


Sk-sidjrdL  C^jriuii 


PARTICIPATING  ORGANIZATION 


International 
.  .  Organization  for 
Standardization 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 
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Relationship  to  NIST  Guidance 


•  NIST  provides  guidance 

•  Risk  Management  Framework 
addresses  controls 
management  (800-37,  800-53, 
et.  al.) 

•  Risk  Hierarchy  forms  the  basis 
for  an  enterprise  risk 
management  program  (800-39) 

•  RMM  maps  to  a  risk 
ecosystem  to  actualize 
and  extend  the  NIST 
guidance 


Risk  Management  Hierarchy 


by  (h*  Risk  Encuthn  Function 

-  EntwpHi*  An^hiHoclura  andSDLCf«-mi 

-  FhilbltiiHjAglltilinptflflMrtfldQfl 


QrganizalLon 


Mission  i  Business  Process 


Information  System 


STRATEGIC  RISK 
FOCUS 
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RMM  Risk  Ecosystem  example 


600-39 

600-53  PM-'S-  ‘‘Risk  Executive  C rit i cal  success 


Software  Engineering  Institute  CarnegieMellon 
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User  Group  2010 
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CERT-RMM  Current  Activities 


Activity 


CERT-RMM  vl.O 
Development 


CERT-RMM 

Coursework 


April  2010  May  2010  June  2010  July  2010 


CERT-RMM 
Capability  Appraisal 
Method 


Other  activities 


August 

2010 


September  October 
2010  2010 


CERT-RMM  vl.O  Technical  Report 


CERT-RMM  A-W  Book 


CERT-RMM  Crosswalk  Update 


Intro  CERT-RMM  (DC)  6-9 


Begin  vl.1  development  cycle 


'  Intro  CERT-RMM  (PIT)  18-21 


CERT-RMM  CAM  Boot-camp 


-  Announce/implement  PCP - 


Intro  CERT-RMM  (DC) 


Intermediate  CERT-RMM 


Publish  CAM  Method  Document 


Develop  formal  licensing  program 


• - Update  PIID  guidance 


-  CERT-RMM  Class  C  Appraisal 


First  CERT-RMM  Class  A  Appraisal 
Fall  2009 


Initiate  M  &  A  Project 


Resiliency  Measurement  Users  Group 


Second  CERT-RMM  Class  A  Appraisal 


M  &  A  Technical  Report 


Software  Engineering  Institute  CarnegieMellon 
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Questions?? 
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Carnegie  Mellon 


RMM  Project  Team  and  Contacts 


Rich  Caralli 

RMM  Architect  and  Lead  Developer 
rcaralli@cert.org 

Lisa  Young 

RMM  Appraisal  Lead  &  Developer 
lry@cert.org 

Richard  E  Barbour 

RMM  Appraisal  Developer 

reb@cert.org 


David  White 

RMM  Transition  Lead  &  Developer 
dwhite@cert.org 

Julia  Allen 

RMM  Developer 
jha@sei.cmu.edu 

Joe  McLeod 

For  info  on  working  with  us 

jmcleod@sei.cmu.edu 


Richard  Lynch  SEI  Customer  Relations 

Public  Relations  —  All  Media  Inquiries  customer-relations@sei.cmu.edu 

public-relations@sei.cmu.edu  412-268-5800 


www.  cert,  org/resiliency 
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This  material  is  considered  SEI-Proprietary  and  is  distributed  by  the  Software  Engineering 
Institute  (SEI)  to  SEI  Staff  ONLY. 

This  material  SHALL  NOT  be  reproduced  or  used  for  any  other  purpose  without  requesting 
formal  permission  from  the  SEI  at  permission@sei.cmu.edu. 

THE  MATERIAL  IS  PROVIDED  ON  AN  “AS  IS”  BASIS,  AND  CARNEGIE  MELLON 
DISCLAIMS  ANY  AND  ALL  WARRANTIES,  IMPLIED  OR  OTHERWISE  (INCLUDING, 

BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR  A  PARTICULAR  PURPOSE, 
RESULTS  OBTAINED  FROM  USE  OF  THE  MATERIAL,  MERCHANTABILITY,  AND/OR 
NON-INFRINGEMENT). 
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Software  Engineering  Institute 


Carnegie  Mellon 


Back-ups 


CERT-RMM  links  to  codes  of  practice 


/ 

/  Codes  of  Practice: 

BS25999-1:2006 
CMMI  vl.2 
CM  Ml  for  Services 
CobiT  4. 1 

!  COSO  ERM 

DRII  GAP 


FFIEC  Handbooks  (Security, 
BCP) 

ISO  20000-1 .2005(E) 

ISO  20000-2:2005(E) 

ISO  24762:2008(E) 

ISO  27001:2005 
NFPA  1600  (2007) 

PCI  DSS  vl.1 


Val-IT 


% 

\ 


\ 

\ 
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GAO-09-835T  report  says: 


An  underlying  reason  for  the  apparent  dichotomy  of  increased  compliance  with 
security  requirements  and  continued  deficiencies  in  security  controls  is  that 
the  metrics  defined  by  OMB  and  used  for  annual  information  security 
reporting  do  not  generally  measure  the  effectiveness  of  the  controls  and 
processes  that  are  key  to  implementing  an  agency  wide  security  program. 

Results  of  our  prior  and  ongoing  work  indicated  that,  for  example,  annual 
reporting  did  not  always  provide  information  on  the  quality  or  effectiveness 
of  the  processes  agencies  use  to  implement  information  security  controls. 
Providing  information  on  the  effectiveness  of  controls  and  processes  could 
further  enhance  the  usefulness  of  the  data  for  management  and  oversight 
of  agency  information  security  programs. 


Software  Engineering  Institute  CarnegieMellon 
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CERT-RMM  timeline 


2003 


2004 


2005 


I 


2006 


2007 


8 

2 

2010 


Best-in-Class  Operations  Roundtable 


Technical  Report:  “Managing  foj  Enterprise  Security ”  n 


"Began  collaboration  with  FSTC 


Commenced  development  ofPRISM:  Process 


Technical  Report?  “Sustaining  Operatic 


iment  Approach  to  Security  Mai 


per 

Jty 


CERT  Resiliency  Engineering  Framework  v0.95  released 


CERT  Resiliency  Engineering  £rameworkjpfO$  5  benchmarking  effort 


Intro  to  CERT^Resilienc^^n^ineerin^Framework  course  pilo\ 


CERT  Resilience  Management  Model  v0.95  re  leas 


^=-  Software  Engineering  Institute  CalTCJIFTTTTOfi 


CERT_  Resilience  Managem  ^nt  Model  vl.O 
CERT-RMM  vl.O  Addison-tolm 


User  Group  2010 

©2010  Carnegie  Mellon  University 


CERT-RMM  and  CMMI-SVC 


Expands  SCON  to  resiliency  perspective 


Shares  an  organizational  focus ,  rather  than  project 


Focus  is  on  high-quality  service  delivery  that  is  resilient 


Model  use  will  identify  additional  synergies 


Software  Engineering  Institute  CarnegieMellon 
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CERT-RMM  process  area  structure 


CERT-RMM  Product  Suite 

Model  artifacts  available  to  begin  an  adoption 
process 


Software  Engineering  Institute  CarnegieMellon 
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CERT-RMM  product  suite 


Product 

Status 

CERT-RMM  Model 

Version  1.0  released;  Technical  Report  released;  individual 
process  areas  released  (a)  www 

CERT-RMM  Capability 
Appraisal  Methodology 

Version  1 .0  to  be  released  in  method  description  document, 
August  2010 

CERT-RMM  Crosswalk 

Version  0.95  published;  Version  1.0  (expanded)  to  be 
published  late  Summer 

Introductory  courses 

Introduction  to  CERT-RMM  (4  days;  offered  4  times/year  in 
Pittsburgh  and  DC) 

Executive  workshops  and  tutorials  available  on  demand 

Advanced  courses 

CERT-RMM  Intermediate  Course  (in  development  for  2011) 
CERT-RMM  CAM  BootCamp  (pilot  scheduled  for  November 
2010) 

CERT-RMM  Role  training  (Coach,  Navigator) 

CERT-RMM  instructor  training 

CERT-RMM  book  publication 


Scheduled  for  publication  in 
November  201 0  by  Addison- 
Wesley 

Includes  full  model  (vl.O)  plus 
adoption  guidance  and 
perspectives  of  real-world  use  of 
the  model 


CERT'  Resilience 
Management  Model 

improving 

/ f  f  Operational 

w  1  -  Jkiitience 

PrDtrjsci 


Rkhwd  A,  CmbIH 


Julia  ]  J.  Allen 
David  W.  Whin: 
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Resilience  measurement  &  analysis 


Area  of  research  growing  out  of  CERT- 
RMM  development 

Focuses  on  the  development  of  adequate 
measures  to  determine  transformation 
of  operational  resilience  management 
system 

Focuses  on  performance  measurement — 
how  well  are  we  doing? 

Includes  both  qualitative  and  quantitative 
measurements 


Measurement  users  group  (RMM  MUG) 
forming — Fall  2010  opportunity  to  join 
a  measurement  cohort  and  share 
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One  RMM  Risk  ecosystem 


•  Incident  Management  and 
Control  (IMC) 

•  Vulnerability  Analysis  and 
Resolution  (VAR) 

•  Compliance  Mgmt.  (COMP) 

•  Technology  Management 
(TM) 

•  Knowledge  and  Information 
Management  (KIM) 

•  Asset  Definition  and 
Management  (ADM) 

•  Service  Continuity  (SC) 


•  Controls  Management  (CTRL) 

•  Enterprise  Focus  (EF) 

•  Monitoring  (MON) 


Software  Engineering  Institute  CarnegieMellon 


Rick  Barbour  CERT  RMM 

10th  CMMI  Technology  Conference  & 

User  Group  2010 

©2010  Carnegie  Mellon  University 


37 


Alignment  with  NIST  Risk  Management  Framework 


RMM  Risk  Eco-System 

Focused  on  operational  risk 
management  process 

Provides  the  basis  to  actualize 
the  NIST  view  of  risk 
management  (e.g.  methods 
to  examine  conditions  and 
consequences  and  link 
assets  to  services) 

Provides  the  basis  for  a 
sustainable,  repeatable, 
efficient  and  measurable  risk 
management  process 


NIST  RMF 

Practical  guidance  for  risk 
assessment  of  IT  systems 
and  application  of  controls 

Provides  foundation  for  the 
development  of  a  threat 
management  program 
based  on  control  selection 
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CERT-RMM  by  the  numbers 
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CERT-RMM  coverage  of  codes  of  practice 


Currently  mapped  to  CERT-RMM:  /n  progress  or  consideration : 


BS25999-1 :2006 
CMMI  vl  .2 
CMMI  for  Services 
CobiT  4.1 
COSO  ERM 
DRII  GAP 


ISO  SE7  Application  Security  Std 

HR1 -Title  9  Voluntary  Standard 
(TBD) 

NIST  standards/FISMA 
provisions 


FFIEC  Handbooks  (Security,  BCP) 
ISO  20000-1 :2005(E) 

ISO  20000-2:2005(E) 

ISO  24762:2008(E) 

ISO  27001:2005 


Documented  in  the  REF  Code 
of  Practice  Crosswalk ,  v0.95R 
to  be  updated  with  release  of 
RMM  version  1.0 


^=-  Software  Engineering  Institute  CarnegieMellon 

~  PCI  DSS  vl.1 
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CERT-RMM  position  in  lifecycle 


Plan 


Design 


Acquire 


Develop  Deploy 


l  l 

I  U 

i _ ! 


t 


CM  Ml- A  CQ 


CMMI-SVC 


DEVELOPMENT 


^Operate 


Decommission 


CERT-RMM 


OPERATION 
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Resiliency  Project  Members 


AMD 

Ameriprise 
Bank  of  America 
Carnegie  Mellon 
Capital  Group 
Citicorp 

Discover  Financial 

EMC 

DRII 

FSSCC  R&D* 


IBM 

JPMorgan  Chase 

Key  Bank 

KPMG 

MasterCard 

Marshall  and  llsley 

NY  Federal  Reserve  Bank* 

PNC  Bank 

US  Bank 

Wachovia 


RMM  codifies  best  practices  for  security 
and  business  continuity  from  world 
leading  organizations  and  numerous 
standards  and  codes 
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Services 

Technology 

Consortium 
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Example:  Asset  Definition  &  Management 


Goals 

Practices 

ADMiSGI  Establish  Organizational 
Assets 

ADM:SG1.SP1  Inventory  Assets 

ADM:SG1.SP2  Establish  a  Common 
Understanding 

ADM:SG1.SP3  Establish  Ownership 
and  Custodianship 

ADM:SG2  Establish  Relationship 
Between  Assets  and  Services 

ADM:SG2.SP1  Associate  Assets  with 
Services 

ADM:SG2.SP2  Analyze  Asset-Service 
Dependencies 

ADM:SG3  Manage  Assets 

ADM:SG3.SP1  Identify  Change 

Criteria 

ADM:SG3.SP2  Maintain  Changes  to 
Assets  and  Inventory 
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Institutionalizing  Asset  Definition  &  Management 


Goals 

Practices 

ADM:SG1  Establish 
Organizational 

Assets 

ADM:SG1.SP1 
Inventory  Assets 

ADM:SG1.SP2 
Establish  a  Common 
Understanding 

ADM:SG1.SP3 

Fstahlish  Ownershin 

and  Custodianship 

ADM:SG2  Establish 
Relationship 

Between  Assets  and 
Services 

ADM:SG2.SP1 
Associate  Assets 
with  Services 

ADM:SG2.SP2 

Analyze  Asset- 
Service 

Dependencies 

ADM:SG3  Manage 
Assets 

ADM:SG3.SP1 

Identify  Change 
Criteria 

ADM:SG3.SP2 

Maintain  Changes  to 
Assets  and  Inventory 

A  managed  process  is: 

•  Governed 

•  Executed  according  to  policy 

•  Employs  skilled  people 

•  Involves  relevant  stakeholders 

•  Monitored,  controlled,  and 
reviewed 

•  Evaluated  for  adherence  to  the 
organization’s  process 
description 

•  Regularly  reviewed  with  senior 
management 
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Practice  example:  ADM. SGI. SP1  -Inventory  Assets 


To  institutionalize  the  performance  of  the  “Inventory  Assets”  practice,  you  must  commit  to 
and  perform  these  supporting  practices: 


Institutionalizing  Factor 


Institutionalizing  Practice 


Governed 


There  is  a  policy  requiring  periodic  asset  inventory  activities;  the  activity 
has  oversight  and  corrective  actions  are  taken  when  necessary 


Employs  skilled  people 


Staff  involved  in  the  practice  have  the  appropriate  skill  levels  and  training 


Involves  stakeholders 


Monitored  and  controlled 


Asset  owners  and  custodians  are  involved;  all  involved  in  protecting  and 
sustaining  the  asset  are  involved 

The  process  is  measured  to  determine  effectiveness.  Examples:  %  of 
assets  inventoried;  #  of  changes  to  inventory  in  a  given  period 


Evaluate  adherence 

Review  with  senior 
management 


The  process  as  performed  is  verified  to  be  aligned  with  the  process 
definition 

Keep  management  informed  on  the  results  of  the  process  and  identify 
and  resolve  issues 
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CERT-RMM  principle  of  convergence 


Operational  resilience  is  directly  affected  by  convergence 
Organizational  mission  is  directly  affected  by  operational  resilience 
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Positioning  CERT-RMM  in  CMMI 


/  X 


\ 


CMMI-ACQ 


i  { 

U«MA£1S!/A-- 

Ns  s 


f  \ 

i  r^rjjymj  ■ 


V  s 

/NS 


l/ses  Process  Areas  from  Core  and  CMMI-DEV 


N  \ 


I - 1 


Shares  connection  in  Service  Continuity  (SCON) 
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SCAM  Pi-based  appraisal  methods 
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ERT-RMM 


Extending  process  improvement  capabilities  to  the  operations  phase  of  the  lifecycle 
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Topics 


> Lockheed  Martin  Aeronautics  Overview 

*  OSP  Deployment  Plan 

*  OSP  Deployment  Process 

*  OSP  Deployment  Event  Planning 

*  Conclusions 


Enterprise  Process  Integration  > 

EPi 
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Lockheed  Martin  Aeronautics 
Overview 


X 


29,000  employees  across  the  company  and  around  the 

world 


International  -  39 


Palmdale,  CA 
3,834 


Johnstown,  PA  - 103 

Clarksburg,  WV  -  78 

Fort  Worth,  TX  ^6^864  ^®enville’  sc  ' 3400 
14^.529 
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Topics 


*  Lockheed  Martin  Aeronautics  Overview 
>OSP  Deployment  Overview 

*  OSP  Deployment  Process 

*  OSP  Deployment  Event  Planning 

*  Conclusions 


Enterprise  Process  Integration  > 

EPi 
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Terminology  -  Organizational 
Standard  Processes  (OSP) 


*  OSP  is  the  set  of  AeroCodes  that  are  used  as  the 
basis  for  establishing  common  development 
processes  across  Aeronautics.  The  OSP  describes  the 
fundamental  processes  that  must  be  incorporated  into 
the  defined  processes  that  are  implemented  by 
functions  and  on  programs  across  Aero. 


Enterprise  Process  Integration  > 
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Terminology  -  Program  Defined 
Process  (PDP) 


*  PDP  is  the  Program  Defined  Process  used  by 
personnel  assigned  to  a  program  when  executing 
work  on  the  program 

-  A  PDP  provides  a  basis  for  planning,  performing, 
and  improving  the  program  tasks  and  activities.  It  is 
tailored  from  the  OSP  to  meet  the  objectives, 
constraints,  and  environment  of  a  specific  program, 
in  accordance  with  the  organization's  process 
tailoring  criteria  and  guidelines. 

*  The  PDP  is  documented  in  the  Program  Adoption  of 
the  OSP  (PAOSP)  for  approval  and  configuration 
control 


Enterprise  Process  Integration 


Enterprise  Process  Integration 

ERf 
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Terminology  -  Tailoring 


*  Tailoring  is  required  for  deletions,  modifications,  and 
substitutions  to  process  requirements,  including 
content  (but  not  format)  of  work  products 

-  Or  when  a  process  requirement  is  inappropriate  for 
a  particular  contract,  a  tailoring  is  requested 

*  Approved  tailorings  are  referenced  in  the  PAOSP 
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OSP  Deployment  Objectives 


*  Ensure  all  Aero  programs  are  utilizing  the  best 
available  processes  to  execute  their  development 
projects 

*  Reinforce  the  use  of  OSP  AeroCodes  to  document  and 
communicate  Aero’s  best  practices 

*  Identify  best  practices  on  programs  for  inclusion  in 
OSP  AeroCodes 

*  Generate  and  verify  a  compliant  set  of  program  plans 
and  processes  tailored  to  meet  business  needs 


Enterprise  Process  Integration  > 

EPi 
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OSP  Deployment  Plan 

•  LM  Aero  Strategy 

-  Bring  current  programs  to  OSP-Compliance 

-  Deploy  OSP  to  all  new  pursuits 

-  Deploy  OSP  changes  to  programs  already  compliant, 
when  required 

•  OSP  Deployment  Approach 

-  Conduct  gap  analysis  events  to  accelerate  development 
of  compliant  program  plans 

-  Identify  gaps  between  OSP  and  program  plans 

-  Establish  set  of  processes  used  on  each  program  based 
on  the  OSP  and  record  them  in  a  PAOSP 

-  Develop/update  and  approve  plans  to  incorporate 
processes,  especially  the  PMP  and  SEMP 


Enterprise  Process  Integration  • 

EPi 
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OSP  Deployment  To  Confirm  A 
Compliant  Program  Baseline 


Inputs 


Existing  OSP 
(Enterprise  Proven 
Practices) 


Existing  Set  of  the 
Program’s  Plans  & 
Procedures 


Outputs^ 


Deploy  OSP; 

Conduct  Gap 
Analysis 

Establish  initial 
PAOSP 

Develop/Implement 
Gap  Closure  Plans 

Submit/Approve 
Tailoring  Requests 

Update/Approve 
Program  Plans 

Approve  PAOSP 

Improve  OSP 


Updated  OSP 
(Enterprise  Proven 
Practices) 


PAOSP  with 
Program’s 
Approved 


OSP-Compliant 
Set  of  Approved 
Program’s  Plans  & 
Procedures 
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Example  of  PAOSP  Spreadsheet 


Result  of  Evaluation 

Domain 

AeroCode 

Number 

Title 

/  /  / •&  /&  A  / 

/  /  /#  A#  / 

/&  /s  /$  A?  /&  J/ 

Description/Objective  /  O  /  v  /  ^  ^  ^  / 

PM 

AC-XXXX 

Capture  Transition 

This  process  outline  the  activities 
and  infromation  required  of  all 
participants  during  the  final  stage 
of  the  pursuit... 

John  Doe 

process 

V 

PM 

AC-XXXX 

Program  Management 

Contract  Close-out 

Tasks 

To  identify  the  responsibilities  of 
the  Program  Manager  and  the 
Program  Leadership  Team  for  the 
Contract  Close-out... 

John  Doe 

process 

V 

Eng 

AC-XXXX 

Build-to-Package 

Process  Control  Manual 
-  Design 

The  Build-to-Package  Process 
Control  Manual  (BTPCM)  -  Design 
describes  the  LM  Aero  BTP  - 
Design  processes  per.... 

James 

Smith 

process 

V 

PM 

AC-XXXX 

Integrated  Cost  and 
Schedule  -  Earned  Value 

Management 

The  Integrated  Costand  Schedule 
( 1  CAS)  -  Earned  Value 

Management  System  (EVMS) 
process  is  the  single.... 

Mary  Jones 

process 

V 

Tech  Pubs 

AC-XXXX 

Technical  Publication 
Development  &  Support 

Process 

The  Technical  Publication 
Development  &  Support  process 
provides  the  means  to  transform 
engineering... 

Carol  Smith 

process 

V 

Enterprise  Process  Integration 
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Topics 


*  Lockheed  Martin  Aeronautics  Overview 

*  OSP  Deployment  Plan 
>OSP  Deployment  Process 

*  OSP  Deployment  Event  Planning 

*  Conclusions 
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OSP  Deployment  Responsibilities 


*  Functions  (Process  Owners) 

-  Assist  development  of,  review  &  approve  program  plans 

-  Work  with  program  to  establish  PDP  including  any 
needed  tailoring 

*  Program  Manager  or  Equivalent 

-  Supported  by  program  Subject  Matter  Experts  (SMEs) 

-  Oversees  the  development  of  program  plans  required  by 
functional  processes 

-  Establishes  the  PDP  including  any  needed  tailoring 

-  Oversees  the  development  and  implementation  of  gap 
closure  plans 

*  Enterprise  Process  Integration 

-  Facilitates  function  and  program  assessment  of  plans 

-  Aids  in  review  and  approval  of  plans  and  tailoring 

-  Drives  deployment  effort  to  conclusion 


Enterprise  Process  Integration  > 
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Gap  Analysis  Review  Steps 

1.  Review  OSP  against  Program  plans  to  identify  any 
differences  (gaps)  -  via  self-assessment  between 
program  POCs  and  functional  SMEs 

2.  Identify  and  address  gaps  between  OSP  and  Program 
plans  via  gap  closure  plans 

Gap:  is  an  identified  non-compliance  to  a  specific  OSP 
AeroCode  or  process  in  an  OSP  AeroCode  which  the 
program/project  intends  to  comply  with  or  tailor. 


3.  Record  results  in  PAOSP  Spreadsheet 

4.  Update,  submit  and  approve  Program  plans 


Enterprise  Process  Integration  > 
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Step  #1  -  Review  OSP  Against 
Program  Plans 


•  Review  the  existing  plans  against  the  applicable 
process  checklist  and/or  AeroCodes  to  identify  gaps 

Or,  if  plans  don’t  exist... 

*  Discuss  program  planning  with  the  applicable  SME 
and  program  manager  to  determine  if  the  steps  in  the 
AeroCode  will  be  followed 


Enterprise  Process  Integration  > 
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Step  #2  -  Identify  &  Address  Gaps 


*  Gap  items  are  typically 

-  Program  actions 

*  Implement  the  OSP  on  the  program 

*  Update  /  develop  required  program  plans  and 
submit  for  approval 

-  Function  actions 

*  Improve  /  update  OSP  (AeroCodes) 

*  Support  programs  to  develop/update  plans 

*  Provide  training 

*  POCs  are  assigned  to  develop  gap  closure  plans 

*  Functional  and  program  leadership  monitor 
implementation  of  gap  plans  until  closure 


Enterprise  Process  Integration  > 

Em 


Copyright  2010  Lockheed  Martin  Aeronautics  Company 


16 


Step  #3  -  Record  Results 


*  Based  on  the  review  of  the  plan  or  program  planning, 
determine  if  the  AeroCode  will  be: 

*  Used  As  Written 

-  Use  or  import  process  asset  without  deletions  or 
substitutions 

*  Tailored 

-  Modify  for  inclusion  with  deletions  or  substitutions 

*  Not  Applicable 

-  None  of  the  AeroCode  applies  based  on  the  program’s 
contract  or  life  cycle 

*  Substitute  with  Program  Implementation 

-  Create  new  process  asset  to  replace  an  AeroCode 


Enterprise  Process  Integration  > 
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Step  #3  -  Record  Results  (cont’d) 


•  Capture  decisions  in  PAOSP  worksheet 

-  As  program  manager  or  delegates  reviews  the  OSP  and 
determines  its  applicability  on  the  program, 
documentation  of  this  decision  is  captured  in  the 
PAOSP  spreadsheet 


-  Authors  of  the  applicable  program  plans  will  use  the 
PAOSP  worksheet  as  guidance  in  creating  /  updating 
their  plan _ 


Result  of  Evaluation 

Domain 

AeroCode 

Number 

Title 

/  /  A  A  A  A 

4  4  4  //  4// 

/ S  Af  An  /At  / 

Description/Objective  /  *5  /  *5*  /  ^  ^  / 

PM 

AC-XXXX 

Capture  Transition 

This  process  outline  the  activities 
and  infromation  required  of  all 
participants  during  the  final  stage 
of  the  pursuit..,. 

John  Doe 

process 

V 

PM 

AC-XXXX 

Program  Management 

Contract  Close-out 

Tasks 

To  identify  the  responsibilities  of 
the  Program  Manager  and  the 
Program  Leadership  Team  for  the 
Contract  Close-out... 

John  Doe 

process 

y 

Enterprise  Process  Integration 
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Step  #4  -  Update  and  Approve 
OSP-Compliant  Program  Plans 


*  Update  existing  program  plan  or  develop  new  program 
plans 

*  Submit  program  plans  to  the  applicable  functional 
SME  for  review  and  comments 

*  Functional  SME  approves  the  plan  (original  and 
updated  versions)  and  any  NA’s 


Enterprise  Process  Integration  > 
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Topics 


*  Lockheed  Martin  Aeronautics  Overview 

*  Enterprise  Common  Process  Management 

*  OSP  Deployment  Plan 

*  OSP  Deployment  Process 
>OSP  Deployment  Event  Planning 

*  Conclusions 
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Enterprise  Process  Integration 


OSP  Deployment  Event  Schedule  Summary 


Day  1:  EPI.  PM.  F&BO,  MM  &  Eng 

-  Opening  Event  Brief  -  Purpose, 
Objectives,  etc. 

-  Program  Management  Plan  (PMP) 

-  IMP  &  IMS 

-  Communication  Plan 


Day  3:  Eng,  PPQA,  &  EPI 

-  SEMP  &  Technical  Planning 

•  Product  Integration 

•  BTP/WPEs 

-  Software  Development  Plan 

-  Measurement  and  Analysis  Plan 

-  Quality  Assurance  Plan 

-  Remaining  OSP  Assessments 


-  Monitor  and  Control  Planning  (PMP) 

-  Resource  Plan 

-  EVMS(PMP) 

-  Estimating  &  Pricing  (PMP) 

-  Strategic  Sourcing  Plan 

-  Subcontract  Mgt  Plan 

-  Configuration  Mgt  Plan 


Day  2:  Eng,  PM  &  F&BO 

-  Data  Mgt  Plan 

-  Risk  &  Opportunity  Mgt  Plan 

-  SEMP  &  Technical  Planning 

•  Requirements  /  WPEs 

•  Verifica  tion  /  Valida  tion 

•  Architecture 

•  Preliminary  &  Detail  Design 

•  Implementation 


Day  4:  All 

-  Create  Event  Outbrief 

-  Conduct  Event  Outbrief 


Note:  If  team  cannot  hold  to  schedule,  then 
parallel  gap  sessions  will  be  conducted 


Lockheed  Martin  Aeronautics  Company 
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Program  OSP  Gap  Analysis 
Objectives 


•  Inventory  the  program  defined  processes  by  dispositioning 
the  “Core”  OSP  AeroCodes 

-  Accept  as  is 

-  Accept  with  tailorings 

-  Not  applicable:  due  to  program  business  need 

-  Substitute  program  process  implementation 

-  Defer:  discuss  during  follow-on  telecon 

*  Determine  what  actions,  if  any,  are  required  to  ensure 
compliance  with  the  LM  Aero  Process  Policy 

-  Program  actions 

-  Core  Function  actions 


Enterprise  Process  Integration  > 

EPi 
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Program  Outcomes 


*  Gap  analysis  event  ends  when... 

-  Core  AeroCodes  in  the  OSP  are  dispositioned 

-  Initial  PAOSP  worksheet  is  completed  and 
tailorings  are  identified 

-  Gap  action  plans  are  developed  and  POCs  assigned 

*  Event  follow-up  ends  when... 

-  Final  PAOSP  worksheet  is  completed  and  tailorings 
are  dispositioned 

-  Any  plan  updates  are  completed  and  approved 

-  Gap  action  plans  are  completed 


Enterprise  Process  Integration  > 

EPi 
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OSP  Deployment  Monitoring 


Initial  OSP  Deployment  Status  Summary 

Program/ 

Project 

Event  Status 

OSP  Assessment 

Program  POC 

Comments 

Planned  , 

Comp 

Assmt's 

Complete 

Tailorings 
Complete  - 
Approved/ 
Total 

Acceptance 

Status 

F-XX 

2007 

2007 

100% 

6/6 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

F-YY 

2007 

2007 

100% 

2/2 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

F-ZZ 

6/2008 

6/2008 

100% 

3/3 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

C-1 

8/2008 

8/2008 

100% 

2/2 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

F-A 

1/2009 

1/2009 

100% 

0/0 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

F-B 

6/2009 

6/2009 

100% 

1/1 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

C-2 

11/2009 

11/2009 

100% 

0/0 

Complete 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

C-3 

12/2010 

12/2010 

25% 

In-work 

Jon  Doe  762-1000 

Deployment  Event 
Scheduled 

Non-Program  Deployment  Events 

Tool  XYZ 

8/2009 

8/2009 

100% 

N/A 

N/A 

Jon  Doe  762-1000 

Initial  OSP 

Deployment  Complete 

Enterprise  Process  Integration  > 

EPi 
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Questions? 
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Agenda 
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•  The  Set-up 

•  How  CMMI®  Gets  Done 

•  Options  for  Implementation 

•  Realize  the  Organization  is  a  System 

•  Implementing  a  Systems  Approach  to  CMMI® 
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•  Populating  the  Systems  Framework 

•  Summary 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 


2 


Approved  for  Public  Release:  Northrop  Grumman  Aerospace  Systems  Case  10-1317  Dated  10/13/10 


The  Set-up 


NORTHROP  GRUMMAN 


•  Its  midnight,  and  a  small  group  of  process  experts  are  burning  the 
candles,  preparing  for  an  up-coming  appraisal 

-  Sound  familiar? 


3 


Approved  for  Public  Release:  Northrop  Grumman  Aerospace  Systems  Case  10-1317  Dated  10/13/10 


How  CMMI  Gets  Done 


NORTHROP  GRUMMAN 


•  A  Common  Method 

-  In  many  organizations,  a  small  cadre  of  process  experts  develops  the  CMMI 
approach  for  their  organization 

•  Map  CMMI  to  processes 

•  Ensure  processes  comply 

•  Identify  and  collect  artifacts 

•  Understanding  the  Evolution  of  the  Common  Method 

-  CMM®  began  as  a  Software  model,  so  naturally  a  Software  group  generally 
comprised  the  cadre  responsible  for  implementation 

-  When  the  CMMI  for  System  and  Software  came  out,  many  organizations 
expanded  the  responsibility  for  implementation  to  include  the  larger 
Engineering  group 

•  The  scope  of  implementation  remained  primarily  within  the  Engineering  disciplines 

-  With  the  IPPD  extension  of  CMMI,  the  potential  for  a  broader  implementation 
outside  the  boundaries  of  Engineering  disciplines  became  a  practical  option  for 
organizations 
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How  CMMI  Gets  Done 


NORTHROP  GRUMMAN 


•  Cultural  Roadblocks  to  an  Expanded  Implementation 

-  Given  the  evolution  of  the  model,  and  its  Engineering-centric  legacy, 
process  experts  within  Engineering  disciplines  typically  remain  at  the 
center  of  implementation  efforts 

•  The  CMM/CMMI  model  went  through  an  evolutionary  expansion  yet  that 
hasn't  necessarily  translated  as  a  change  in  the  size  and  make-up  of  the 
implementation  team 

-  The  Engineering-centric  cadre  remains  relatively  constant  in  many 
organizations 

-  The  result: 

•  A  broad  organizationally  based  model  is  often  implemented  by  a  small 
specialized  group 
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Options  for  Implementation 


NORTHROP  GRUMMAN 


•  Overcoming  the  Roadblocks 

-  How  can  a  small  cadre  effectively  implement  a  broad  model  so  that  the 
organization  at  large  will  understand,  embrace,  and  institutionalize  the 
implementation? 

•  Option  1: 

-  Restrict  the  organizational  scope  of  the  model  primarily  to  Engineering 
activities 

•  Option  2: 

-  Acquire  the  necessary  resources  to  expand  the  scope  beyond  Engineering 
to  include  other  departments 

»  E.g.  Program  Management,  Sub-Contract  Management,  or  Human 
Resources 
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Options  for  Implementation 


NORTHROP  GRUMMAN 


•  Option  1: 

-  Consider  the  consequences  when  the  scope  is  restricted  to  Engineering 

•  Although  the  approach  is  easier  for  an  Engineering-based  cadre  to 
manage,  there  are  downsides: 

-  Project  management  functions  (e.g.  PP,  PMC,  I  PM,  RSKM,  CM,  SAM,  DAR) 
may  be  limited  to  technical  management 

»  This  is  likely  to  exclude  important  higher  level  non-engineering 
functions 

-  The  restricted  approach  leaves  an  organization  with  the  inability 
to  capture: 

»  Program  Management  plan 

»  Non-technical  risks  related  to  program  execution 

»  Staffing  resources  associated  with  contract  management,  customer 
interfaces,  and  program  management 
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Options  for  Implementation 


NORTHROP  GRUMMAN 


•  Option  1: 

-  Consequences  -  Continued 

•  Other  disciplines  within  the  enterprise  fail  to  see  the  value  of  the 
CMMI  model  when  it  is  implemented  with  an  engineering-centric 
approach 

-  They  view  the  model  as  something  important  only  to  engineering 
(engineering  owns  it)  and  of  no  real  use  to  the  program  at  large 

•  The  Result: 

-  A  restricted  view  of  CMMI  applicability 

-  This  may  be  a  viable  option  for  some  organizations,  but  not  all 
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Options  for  Implementation 


NORTHROP  GRUMMAN 


•  Option  2: 

-  Consider  expanding  the  scope  beyond  Engineering 

•  Although  the  approach  is  more  challenging  for  an  Engineering-based  cadre  to 
manage,  there  are  upsides: 

-  Provides  a  more  realistic,  inclusive  view  of  the  Enterprise 

-  The  payoff  is  worth  the  effort 

»  It  results  in  a  more  robust  implementation  of  the  model,  involving  a 
greater  representative  cross-section  of  the  enterprise 

•  Engineering  typically  provides  the  CMMI  knowledge-base  and  the  personnel 
experienced  in  pulling  appraisals  together 

-  The  expanded  approach  requires  bringing  in  players  from  other 
departments,  so  how  does  engineering  steer  this  effort  and  involve  these 
players? 

•  The  answer 

-  You  trick  them 

»  Develop  methods  and  management  support  to  solicit  assistance  from 
departments  outside  Engineering 
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Realize  the  Organization  is  a  System 


NORTHROP  GRUMMAN 


•  Once  the  decision  is  made  to  employ  an  expanded  approach  beyond 
Engineering,  another  decision  must  be  made 

-  What  will  be  scoped  in  and  out  of  the  expanded  model  implementation? 

•  What  are  the  functional  elements  of  the  "scoped"  organization 

-  Organizational  charts  and  charters  can  help  determine  who  has  the  mandate  for 
these  functions 

•  Example:  the  CMMI  model  addresses  resources/facilities  (e.g.  PP  SP2.4  &  I  PM 
SP1.3)  and  organizational  training  (OT) 

-  Determine  which  functional  elements  within  the  organization  typically 
handle  those  areas. 

•  By  viewing  the  organization  as  a  system,  the  inclusion  of  critical 
organizational  elements  can  be  determined  to  satisfy  the  model 

-  Example:  including  Program  Management  and  Global  Supply  Chain  can  be  useful  in 
satisfying  PP,  PMC,  I  PM,  and  SAM 
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Implementing  a  Systems  Approach  to  CMMI 


NORTHROP  GRUMMAN 


•  An  effective  approach  to  encourage  participation  of  non-Engineering 
organizations 

-  Consider  techniques  to  elicit  a  sense  of  ownership  of  parts  of  the  model  among 
other  disciplines 

•  Develop  an  initial  mapping  of  the  "owning"  organizations  to  the  CMMI 
processes 

•  Identify  the  organizational  entities  (think  system  components)  that  might 
"own"  portions  of  the  CMMI  model  (think  requirements) 

•  Identify  whether  the  organizational  entities  have  a  "project"  or  "organizational" 
responsibility  in  relation  to  the  process  area 

-  This  will  tell  you  where  artifacts  will  come  from 

•  For  each  "owning"  organization  identify  the  CMMI  processes  that  they  own 

•  Work  through  executive  management  to  identify  process  subject  matter 
experts  (SMEs)  in  each  discipline  to  assist  in  the  effort 

•  Hold  meetings  with  the  organizational  SMEs  to  secure  their  comments  and 
concurrence  on  the  mapping 
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Identifying  the  Components  of  the  System 


NORTHROP  GRUMMAN 


The  CMMI  model  (process  areas  and 
practices)  can  be  viewed  as  the  allocated 
requirements 


p 

o 


Organizational  departments  can  be  viewed 
as  the  components  of  the  System 


Indicates  that  activity  is  primarily  a  project  activity 

Indicates  that  activity  is  primarily  a  functional  (organizational)  activity 


LEVEL  2 

PROJECT  PLANNING 

Subject  Matter 
Expert 

SYS 

SW 

T&E 

LOG 

HW 

QA 

Process 

Grp 

Eng 

Mgmt 

Global 

Supply 

Chain 

Supplier 

Qual 

Prog 

Mgmt 

SG-1  Establish  Estimates 

SP-1.1 

Estimate  the  scope  of  the  project 

P 

P 

P 

SP-1.2 

Establish  estimates  of  work  product 
&  task  attribute 

P 

P 

P 

P 

P 

P 

SP-1.3 

Define  project  life  cycle 

P 

P 

P 

P 

SP-1.4 

Determine  estimates  of  effort  &  cost 

P 

P 

P 

P 

P 

P 

P 

SG-2  Develop  a  Project  Plan 

SP-2.1 

Establish  the  budget  &  schedule 

P 

P 

SP-2.2 

Identify  project  risks 

P 

P 

P 

SP-2.3 

Plan  for  data  management 

P 

SP-2.4 

Plan  for  project  resources 

P 

P 

P 

P 

P 

P 

P 

SP-2.5 

Plan  for  needed  knowledge  &  skills 

P 

P 

P 

P 

P 

P 

P 

SP-2.6 

Plan  stakeholder  involvement 

P 

P 

SP-2.7 

Establish  the  project  plan 

P 

P 

P 

P 

P 

SG3  -  Obtain  Commitment  to  the  Plan 

SP-3.1 

Review  plans  that  affect  the  project 

P 

P 

SP-3.2 

Reconcile  work  &  resource  levels 

P 

P 

SP-3.3 

Obtain  plan  commitment 

P 

P 
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Identifying  the  Components  of  the  System 


NORTHROP  GRUMMAN 


The  Systems  approach  identifies  which  requirements 
are  satisfied  by  the  project  and  which  are  satisfied 
by  the  organization 


P  Indicates  that  activity  is  primarily  a  project  activity 
O  Indicates  that  activity  is  primarily  a  functional  (organizational)  activity 


SUPPLIER  AGREEMENT  MANAGEMENT 

Subject  Matter 
Expert 

SYS 

SW 

T&E 

LOG 

HW 

QA 

Process 

Grp 

EM 

Global 

Supply 

Chain 

Supplier 

Qual 

Prog 

Mgmt 

SG-1  Establish  Supplier  Agreements 

SP-1.1 

Determine  acquisition  type 

P 

P 

P 

P 

SP-1.2 

Select  suppliers 

P 

P 

p 

P 

P 

SP-1.3 

Establish  suppler  agreements 

P 

P 

p 

P 

P 

SG-2  Satisfy  Supplier  Agreements 

SP-2.1 

Execute  the  supplier  agreement 

P 

P 

P 

P 

P 

SP-2.2 

Monitor  Selected  Supplier 
Processes 

P 

P 

P 

p 

P 

P 

SP-2.3 

Evaluate  Selected  Supplier  Work 
Products 

SP-2.4 

Accept  the  acquired  product 

P 

p 

P 

P 

SP-2.5 

Transition  products 

P 

P 

p 

P 
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Identifying  the  Components  of  the  System 


NORTHROP  GRUMMAN 


The  Systems  approach  identifies  the  subject  matter 
expert  (SME)  for  each  PA  and  practice 


P  Indicates  that  activity  is  primarily  a  project  activity 

O  Indicates  that  activity  is  primarily  a  functional  (organizational)  activity 


LEVEL  3 

TECHNICAL  SOLUTION 

Subject  Matter 
Expert 

SYS 

SW 

T&E 

LOG 

HW 

QA 

Process 

Grp 

EM 

Global 

Supply 

Chain 

Supplier 

Qual 

Prog 

Mgmt 

SG-1  Select  Product-Component  Solutions 

SP-1.1 

Develop  alternative  solutions  and 
selection  criteria 

P 

SP-1.2 

Select  product-component  solutions 

P 

SG-2  Develop  the  Design 

SP-2.1 

Design  the  product  or  product 
component 

P 

P 

SP-2.2 

Establish  a  technical  data  package 

P 

P 

P 

SP-2.3 

Design  interfaces  using  criteria 

P 

P 

SP-2.4 

Perform  make,  buy,  or  reuse 
analyses 

P 

P 

P 

P 

P 

P 

SG-3  Implement  the  Product  Design 

SP-3.1 

Implement  the  design 

P 

P 

SP-3.2 

Develop  product  support 
documentation 

P 
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Identifying  the  Components  of  the  System 


NORTHROP  GRUMMAN 


P  Indicates  that  activity  is  primarily  a  project  activity 
O  Indicates  that  activity  is  primarily  a  functional  (organizational)  activity 


ORGANIZATIONAL  PROCESS  DEFINITION 

Subject  Matter 
Expert 

SYS 

sw 

T&E 

LOG 

HW 

QA 

Process 

Grp 

EM 

Global 

Supply 

Chain 

Supplier 

Qual 

Prog 

Mgmt 

SG-1  Establish  Organizational  Process  Assets 

SP-1.1 

Establish  standard  processes 

0 

0 

0 

0 

9-p 

0 

0 

0 

0 

SP-1 .2 

Establish  life-cycle  model 
descriptions 

o 

o 

0 

SP-1.3 

Establish  tailoring  criteria  and 
guidelines 

0 

0 

0 

0 

SP-1 .4 

Establish  the  organization's 
measurement  repository 

0 

0 

0 

SP-1 .5 

Establish  the  organization's  process 
asset  library 

0 

0 

0 

SP-1 .6 

Establish  work  environment 
standardsprocess  asset  library 

0 

0 

o 

SG-2  Enable  IPPD  Management  | 

SP-2.1 

Establish  empowerment 
mechanisms 

0 

0 

0 

SP-2.2 

Establish  rules  &  guidelines  for 
integrated  teams 

0 

SP-2.3 

Balance  team  and  home 
organization  responsibilities 

0 

0 

0 

o 

The  Systems  approach  identifies  which  requirements  are  allocated  to  which 
components  (organizational  departments),  and  identifies  whether  it  is  an 
organizational  or  project  function 
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Populating  the  System  Framework 
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•  Mapping  the  Documentation  to  the  Model  (Engineering  and  Non- 
Engineering) 

-  Work  with  designated  process  SMEs  from  each  discipline/department  to  identify  the 
procedures  that  map  to  process  areas  and  practices 

-  Work  with  the  SMEs  to  identify  specific  passages  in  a  document  that  address  the 
practice 

-  Assist  SMEs  in  gap  analysis  to  identify  any  document  deficiencies  which  require 
remedial  work  to  satisfy  the  practice 

-  Work  with  SMEs  to  identify  the  work  products  their  processes  produce,  which 
correspond  to  the  “expected  artifacts”  cited  in  the  model 

-  Solicit  SME  assistance  in  the  actual  mapping  of  their  procedures  and  expected 
artifacts  into  the  CMMI  process  implementation  indicator  (Pll) 

-  Solicit  SME  assistance  in  collecting  project  specific  artifacts,  and  organizing  them 
into  specified  repositories 

-  Work  with  SMEs  to  identify  the  appraisal  interview  candidates 

-  The  SMEs  become  points-of-contact  to  resolve  findings  generated  from  appraisals 
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Summary 
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•  Benefits:  Tangible  and  Intangible 

-  Tangible 

•  The  expanded  approach  results  in  robust  implementation  of  the  CMMI  model, 
involving  critical  departments  and  management  functions  outside  engineering 

•  The  expanded  implementation  safeguards  against  “flag  chasing”  approaches 
that  rely  on  convenient,  sub-optimized  mappings 

-  Intangible 

•  Process  subject  matter  experts  (SMEs)  from  participating  disciplines  gain 
exposure  to  and  an  appreciation  of  the  benefits  of  CMMI 

-  They  become  process  advocates,  carrying  the  message  of  the  model’s 
usefulness  back  to  their  organization 

-  Communicating  the  benefits  of  CMMI  to  non-Engineering  departments 
uninvolved  in  the  appraisal  process  would  likely  represent  an 
insurmountable  challenge  for  a  small  cadre  of  Engineering  personnel 
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Background 


•  Lockheed  Martin  Aeronautics  measurement  process 
improvements 

-  SEI  CMM  Level  3  for  software  -  1993 

*  SW  Standard  Metrics 

-  SEI  CMM  Level  4  for  software  -  1999 

*  SW  Quantitative  Management 

-  SEI  CMMI  Level  3  at  company  level  -  2007  &  2010 

*  Company  wide  measurement  process 
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Problem 


•  How  to  deploy  the  company  Measurement  and 
Analysis  process  across  multiple  programs  and 
geographical  sites? 

-  Institutionalize  the  M&A  process 

*  Georgia,  Texas,  and  California 

-  Incorporate  M&A  planning  into  the  program 
planning 

-  Identify  and  deploy  M&A  related  training 

-  Programs  already  doing  most  of  M&A  process 

Change  company  culture 


Enterprise  Process  Integration  > 

EPi 
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Approach 


*  Take  the  Measurement  and  Analysis  process  to 
the  programs 

-  M&A  planning  workshops  on  site  with  the 
programs 

-  Facilitated  by  small  team  of  M&A  experts 

-  Pre-populated  M&A  planning  templates 

-  M&A  process  awareness  training  prerequisite 
for  participants 

-  Met  with  program  management 

-  Include  current  program  SMEs  and  middle 
management 


Enterprise  Process  Integration  > 

EPi 
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Approach  (cont.) 


*  Objective:  Develop  a  70-80%  complete  M&A  Plan 
for  the  program  by  end  of  3  day  workshop 

*  Preparation 

-  Schedule  workshop  with  program  POC 

•  Coordinate  appropriate  participants 

-  Pre-populated  plan  template  with  standard 
content 

•  Boiler  plate  material 

•  General  measurement  traceability  data 

•  Measurement  specification  data  for  standard 
measures 

•  Standard  reference  to  M&A  training 

•  Place  holders  for  required  content 


Enterprise  Process  Integration  > 

EPi 
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Approach  (cont.)  - 

•  Objective:  Develop  a  70-80%  complete  M&A  Plan  for  the 
program  by  end  of  3  day  workshop 

•  Onsite 

-  Obtain  buy-in  from  program  manager 

-  Identify  current  Information  Needs  with  middle 
management 

-  Review  current  measures  with  program  SMEs 

•  Match  to  the  Information  Needs 

•  Identify  any  gaps  or  duplications 

-  Load  program  data  in  organizational  database 

-  Generate  plan  tables  from  the  database 

-  Identify  and  fill  in  program  specific  information  in  the 
plan 

-  Identify  remaining  tasks  for  program  POC 
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Workshop  activities 


M&A  Team 
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Workshop  activities  (cont.) 


Program 
Mid  Mgmt 


Program 

SMEs 


Notice  this  frees  up  the  program  to  continue  working 


Program  Mgr 
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Experiences 


*  Programs  appreciated  the  help 

-  Going  to  the  program  minimized  impact  to  the  program 

*  Distributed  planning  experience  across  company 

-  M&A  Team  could  take  best  practices  &  lessons  learned 
from  program  to  program 

*  Planning  made  workshop  go  smoother 

-  Clear  expectations  and  objectives 

*  Flexibility  in  execution  fit  better  with  program  availability 

-  Meet  with  management  in  a  CR  or  go  to  their  offices 

*  Meet  with  program  manager  helped  gain  buy  in  and  visibility 
of  M&A  planning 

*  Many  times  program  had  been  requested  to  improve  their 
measures  just  before  we  arrived 
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Distribution  of  Lessons  Learned 


Program  Z 


m&A  PMrmjrjy 


M&ATeam  &  Plan  Template 


Enterprise  Process  Integration  ■ 
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Experiences  (cont.) 

•  Program  follow  up  to  complete  the  plan  sometimes  became 
low  priority  after  we  left 

-  Took  long  time  to  complete 

•  Program  personnel  assigned  to  complete  plan  sometimes 
were  re-assigned 

•  Program  later  did  not  recognize  M&A  planning  as  significant 
portion  of  program  planning 

•  Program  didn’t  want  to  include  all  measures  in  their  plan 

•  Program  didn’t  use  the  plan  to  mange  their  measures  with 

-  tendency  to  do  it  like  they  always  have 

•  Difficult  identifying  various  reporting  mechanisms  using 
measures 


Enterprise  Process  Integration  > 

EPi 
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Resources 


*  Resources: 

-  Pre-populated  M&A  plan  template 

*  Common  objectives,  information  needs  and 
measures 

*  Boiler  plate  information 

*  Training  references 

*  References  to  standard  process 

-  Measurement  Model  Database 

*  Specification  data  for  standard  measures 

*  Common  objectives  and  information  needs 

*  Able  to  generate  tables  for  M&A  plan 


Enterprise  Process  Integration  > 

EPi 
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Results 


*  Began  culture  change  about  use  of  measurement 

-  Everyone  uses  measurement  but  they  don’t 
plan  how  they  use  them  or  link  them  to  specific 
information  needs 

-  Understand  the  need  to  document  analysis  and 
actions  in  preparation  for  management 
consumption 

-  Concept  to  plan  their  measurement  activities 

-  Expanded  awareness  of  M&A  infrastructure 


Enterprise  Process  Integration  > 

EPi 
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Summary 


*  Institutionalization  of  the  measurement  and 
analysis  process 

*  Culture  change 

*  Take  the  process  to  the  programs 

*  Assisted  the  programs  in  measurement  planning 


Enterprise  Process  Integration  > 

EPi 
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Contact  Information 


•  William  H.  Golaz 

-  LM  Fellow 

-  Lockheed  Martin  Aeronautics 

-  817-935-5701 

-  William.h.golaz@lmco.com 


Enterprise  Process  Integration 


Enterprise  Process  Integration 

ERf 
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NORTHROP  GRUMMAN 


What  Does  HONEST  High 
Maturity  Behavior  Look  Like? 


CM  Ml  Conference 

November  15-18,  2010 


Diane  Mizukami-Williams 
Northrop  Grumman  Corporation 


Abstract  11177 


Objectives 


NORTHROP  GRUMMAN 


•  Intended  for  people  who  are  new  to  high  maturity. 

•  Shows  how  inter-connected  the  high  maturity  process  areas  are. 

•  Stresses  how  using  high  maturity  can  result  in  success,  even  in  your 
personal  lives 

•  Will  use  personal  life  example  to  help  you  understand  basic  principles. 
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NORTHROP  GRUMMAN 


Northrop  Grumman  Information  Systems  (IS)  Sector 


IS  Sector 

•  7,000  contracts 

•  32,000  employees 


Products  and  Services 

Mission  support 
Cybersecurity 
Enterprise  applications 
Command,  control,  and  communications 
IT  &  network  infrastructure 
Management  &  engineering  services 
Intelligence,  surveillance,  &  reconnaissance 


>dcce& 


denied 


CMMI  Appraisals 

•  Over  80  organizations  (over  250  projects)  appraised  at  Level  3  or  higher 

•  Completed  32  Level  5  appraisals 


Almost  a  Personal  Improvement  Presentation 


NORTHROP  GRUMMAN 


to  LOOK  INSIDE  I 

bvER  i  ~\  nmtnnw  souilf 


PEOPLE 


Powerful  Lessons  "2^" 
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Stephen  R.  Covey 


Book  contains  7  principles: 

•  Principles  of  Personal  Vision 
(OPM) 

•  Principles  of  Personal 
Management  (QPM) 

•  8th  new  habit:  Principles  of 
Taking  High  Maturity  Seriously 


Even  self  improvement  textbooks  use  high  maturity  concepts. 


NORTHROP  GRUMMAN 


High  Maturity  Process  Areas  Should  NOT  Be  Like  This 


Railroad 

Tracks 

Never 

Connect 


Should  not  do  each  process  area  in  isolation  and  pop  out  with  a  level. 


High  Maturity  Process  Areas  Should  Be  Like  This 


NORTHROP  GRUMMAN 


Dot-to-dot  where  high  maturity  process  areas  continually 
connect  to  one  another,  e.g.,  OPP  to  QPM  to  CAR  to 
OPP  to  OPM  to  QPM  to  OPP  .  .  . 


High  maturity  continuously  bounces  back-and-forth  between  dots. 


Examples  of  Not  Connecting  the  Dots 


NORTHROP  GRUMMAN 


•  Baselines  and  models  (OPP)  are  not  used  by  projects  (QPM) 

•  Making  improvements  (OPM)  that  are  not  based  on  an  analysis 
of  existing  performance  (OPP) 

•  Managing  your  project  (QPM)  without  an  understanding  of  your 
current  performance  (OPP)  Note  that  baselines  and  models  can 
be  at  a  project  level. 

•  Not  using  what  projects  have  learned  (CAR)  to  make 
organizational  improvements  (OPM) 

•  Not  using  performance  measures  (QPM)  to  help  figure  out  the 
root  cause  of  performance  problems  (CAR) 

•  And  the  list  goes  on  ... 
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For  high  maturity,  some  of  this  behavior  is  expected. 


Personal  Example  is  Money 


9 


NORTHROP  GRUMMAN 


Example  shows  how  I 
manage  money  using  high 
maturity  concepts 

Use  of  high  maturity  concepts 
helped  to  reach  personal 
goals  (business  objectives) 

Many  aspects  of  my  personal 
example  can  be  used  to 
manage  spending  in 
organizations 


When  M&A  Started 


NORTHROP  GRUMMAN 


•  Moved  away  from  home  at  19  years  old  in  1978  with  $600. 

•  Worked  full  time  at  TRW,  Inc.  in  the  graphic  art  department. 

•  For  no  particular  reason,  immediately  started  tracking  spending. 

•  Collected  barely  any  metrics  (kindergarden  M&A) 
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M&A  101  :  Baby  Metrics 


NORTHROP  GRUMMAN 


You  have  to  start  collecting  data,  even  if  you  start  off  with  baby  M&A. 
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M&A  Evolved  Quickly 


By  the  end  of  1979,  M&A  quickly  evolved  to  log  more  details. 
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Only  interested  in  total  spending  (the  process,  not  subprocesses) 
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M&A:  Grown  Up  Metrics 


NORTHROP  GRUMMAN 
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As  you  get  used  to  M&A,  you  “grow  up”  and  gather  more  details. 


M&A  Went  Overboard 


NORTHROP  GRUMMAN 


•  Soon  after,  obsessive  compulsive  disorder  kicked  in. 
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Logged  a 
penny.  Even 
started  to  log 
when  I  found  a 
penny  in  the 
parking  lot. 


•  As  you  collect  data,  you  may  start  to  realize  you  need  more. 
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M&A:  Obsessive  Compulsive  Disorder 


NORTHROP  GRUMMAN 


The  point  when  you  start  to  care  usually  leads  to  high  maturity. 
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The  “Event”:  My  First  OPM  Business  Objective 


NORTHROP  GRUMMAN 
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“Three’s  Company”,  i.e.,  pain,  led  to  my  first  “business  objective” 


OPM  “Business  Objective” 


NORTHROP  GRUMMAN 


Buy  a  house  in  Los  Angeles  before  any  of 
my  friends  by  21  years  old. 


0  ^ 
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OPM:  Vision  to  Buy  a  House  by  21  Years  Old 


NORTHROP  GRUMMAN 


With  OPM,  you  have  a  vision  to  strive  for  “Principles  of  Personal  Vision” 
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QPM:  Performance  Objectives 


NORTHROP  GRUMMAN 


•  Process  performance  objectives: 

-  Increase  my  hourly  rate  as  quickly  as  possible 

-  Control  my  spending 

•  Both  are  measurable  objectives. 
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QPM:  Monitored  Performance 


NORTHROP  GRUMMAN 


•  Continually  tried  to  understand  and  control  spending. 

•  Graphed  typical  spending,  actual  spending,  income,  and  special 
causes  and  tried  to  reduce  variation  in  typical  spending. 

•  Graph  shows  typical  spending  eventually  stabilized. 
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QPM:  Used  Metrics  to  Control  Spending 


NORTHROP  GRUMMAN 


Controlling  spending  includes  stabilizing  and  reducing  variation. 
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OPM:  Achieved  Business  Objective 


NORTHROP  GRUMMAN 


By  age  21  (1980),  went 
from  $4. 50/hr  to  $13. 50/hr 
(7  graphic  art  companies) 

Worked  56  hours  per  week 
at  $20. 25/hr  (overtime  pay) 

Continued  to  track  and 
control  spending  (QPM) 

Significant  measureable 
improvement  towards 
meeting  my  business 
objective  (OPM) 


o  ^ 


Bought  first  house 
at  22  years  old 
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OPM:  Achieved  Business  Objectives 


NORTHROP  GRUMMAN 


Seeing  tangible  measurable  improvements  motivates  people. 
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OPP:  Evolved  to  Subprocess-Level  Knowledge 


NORTHROP  GRUMMAN 


•  By  198X,  I  didn't  have  enough  visibility  into  spending  so  I  broke 
the  "spending  money"  process  into  subprocesses  by  categories, 
e.g.,  food,  clothes,  entertainment,  etc. 

•  Entered  spending  into  Quicken/Excel  to  understand  typical 
spending  patterns  and  trends 


Last  2  years,  spent 
a  total  of  $225  on 
clothes,  which 
includes  shoes, 
underwear,  and 
accessories  (so, . . . 

did  I  learn  to 
control  spending?) 


Clothes 
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OPP:  Understood  Actual  Performance 


NORTHROP  GRUMMAN 


Evolved  to  understand  central  tendency  and  range  for  subprocesses. 
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QPM:  Used  Measures  to  See  Improvements 


NORTHROP  GRUMMAN 


•  Used  OPP  baselines  to  predict  and  improve  QPM  performance 

•  Understand  which  expenses  I  have  control  over 

•  Some  subprocesses  can  be  controlled  through  improvements, 
others  cannot 
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QPM:  Took  Action  to  Meet  Objectives 


NORTHROP  GRUMMAN 


Some  process  changes  have  a  visible  shift.  You  can  alter  your  future. 
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OPM:  Latest  Business  Objective 


NORTHROP  GRUMMAN 


In  2002,  came  up  with  a  new  business  objective,... 

early  retirement  at  55 
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OPM:  Vision  to  Retire  at  Age  55 


NORTHROP  GRUMMAN 
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Business  objectives  should  always  evolve. 


OPP:  Created  Mortgage  Model 


NORTHROP  GRUMMAN 


•  My  husband  also  has  the  same  business  objective  to  retire  at  55. 

•  He  created  an  OPP  mortgage  model  and  did  "what-if"  analysis. 

•  He  refinanced  several  times,  paid  additional  principle  each 
month,  and  at  age  49  in  2010,  paid  off  his  $700,000  and 
$200,000  homes. 
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OPP:  Created  SEI  Intro  to  CMMI  Training  Model 


NORTHROP  GRUMMAN 


•  Part  of  my  retirement  plans  is  to  start  teaching  the  SEI 
Introduction  to  CMMI-DEV  VI. 3  course  publicly  in  Los  Angeles 
on  the  weekends,  starting  in  Jan  2011. 

•  Built  a  model  for  "what  if"  analysis  for  teaching. 


Number  of  Students: 
Number  of  Days: 

Student  Fee: 

Room  Fee  (5150  or  5350): 


Profit: 


$887.92 


Per  Student  Costs 

Per  Student 

Total 

Total 

Breakfast  [2  Days) 
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Lunch  (2  Days) 
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5240.00 
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NORTHROP  GRUMMAN 


OPP:  Use  Models  Regardless  if  They  are  CMMI  Compliant 


”  C:\EJW\MYMISC\PAYMEINTS.EXE 

■  _|n|x| 

Pay 

^  ,  estimating 

•*m 

Loan  Amount:  ■*  . 148.00 

Term:  120  .  -'ths 

<10  yrs  0  nos) 

Interest  Ra  /:  4. 

500 

Standard 

aym  t : 

1525.02 

Stan 

da  Payment 

Estim 

ed  Payment 

Month 

Principal 

I  cerest 

Balance 

Payr  ,t 

Princii 

1  Interest 

Balance 

73 

1274.23 

50.79 

65602.32 

21  .00 

2027.0 

72.94 

17424.54 

74 

1279.01 

M6.01 

64323.31 

*■  J0.00 

2034.6i 

65.34 

15389.88 

75 

1283.81 

24 1.21 

63039.50 

.100.00 

2042. 2! 

57.71 

13347.59 

76 

1288.62 

236.40 

61750.88 

2100.00 

2049.9! 

50.05 

11297.65 

77 

1293.45 

>31.57 

60457 . / 

2100.00 

2057.6! 

42.37 

9240.01 

78 

1298.30 

*26 .72 

59159  3 

2100.00 

2065. 31 

34.65 

7174.66 

79 

1303.17 

21.85 

578r  .95 

2100.00 

2073.  J 

26.90 

5101.57 

80 

1308.06 

V6.96 

5f  .  / . 90 

2100.00 

2080 .  f 

19.13 

3020.70 

81 

1312.96 

.  2.05 

r  .34.93 

2100.00 

2088  2 

11.33 

932.03 

82 

1317.89 

2,  .13 

,3917.04 

935.52 

932  13 

3.50 

0.00 

83 

1322.83 

20.  19 

52594.21 

0.00 

.00 

0.00 

0.00 

84 

1327.79 

197. 

51266.42 

0.00 

J.00 

0.00 

0.00 

Total  Standard  Payments:  lb 

82 .21 

Total  F' 

.mated  Payments:  171035.52 

Pay-Off 

Time:  120 

mos  <10  yrs 

Pa-  ' 

lime:  82 

mos  <  6  yrs 

10  mos) 

(New  Lo 

ad  Save  uRi 

nt  Forward 

Back  Amour 

it  Interes 

t  Payment 

Term  Helo  Q 

uit> : 

Number  of  Students: 
Number  of  Days: 
Student  Fee: 

Room  Fee  ($150  or  $3| 

Profit: 

Per  Student  Costl 


Breakfast  (2  Days) 


Lunch  (2  Days) 


Drinks  (3  Days 


Snacks  (3  Days] 


FedEx  Kinkos 


I  jmim  \  m 

Per  Student 

tAi 

Total 

$10.00 

$120.00 

$13.00 

Ma.2T 

$156.00 

$5.00 

^K5.49 

$90.00 

$2.95 

$3.24 

$53.10 

$40.00 

$240.00 

nn  nn \ - 

Academics  will  say  BAD  DIANE,  those  models  didn't  use  historical 
data  so  they  are  no  good  !!! 

If  a  model  helps  you  make  better  decisions  towards  meeting 
your  business  objectives,  that’s  all  that  should  matter. 
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OPP:  Created  OPP  Models 


NORTHROP  GRUMMAN 
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“All  models  are  wrong,  but  some  are  useful”,  George  Box 


NORTHROP  GRUMMAN 


CAR:  What’s  Still  Wrong  With  My  Spending 


Problem  was  I  still  wasn't  happy  with  my  spending,  but  didn't 
know  how  to  fix  it 

In  2010,  decided  to  do  a  CAR,  and  analyze  spending  in  more 
detail,  and  figure  out  the  root  cause  of  some  wasteful  spending 

Analyzed  the  utilities  subprocess  baseline  (OPP),  which  had  been 
ignored  for  years  to  see  if  there  was  waste. 


Gas 


Water 


Cable  TV 


NetFlix 


Phone 


CAR:  Spending  Improvements  per  CAR  Analysis 


NORTHROP  GRUMMAN 


•  CAR  identified  3  corrective  actions: 


After  many  years  with  Time  Warner,  switched  to 
Verizon  saving  approximately  $30  a  month. 


After  many  years  renting  3  DVDs  at  a  time  from 
NetFlix,  switched  to  2  DVDs  at  a  time 


Made  my  husband  pay  me  half  for  NetFlix 
(he  didn't  like  that) 


•  CAR  analysis,  coupled  with  3  fixes,  helped  my  QPM  objectives 
which  was  to  control  spending 
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CAR:  Fixed  Utilities  Problem 


NORTHROP  GRUMMAN 
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Use  CAR  when  you  know  it’s  broken  but  don’t  know  why. 


OPM:  Where  Are  We  Now  for  Retiring  at  55? 


NORTHROP  GRUMMAN 


After  paying  off  both  homes  in  2010,  my  husband  bought  our 
retirement  home  in  the  mountains  of  Los  Angeles  at  3.5%  interest, 
0  points,  on  1  acre  at  age  49.  Escrow  will  close  Nov  24,  2010. 
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Why  All  the  Bragging? 


NORTHROP  GRUMMAN 


Because  I  want  you  to  beat  me  up  in  the 
parking  lot  so  I  finally  get  time  off  from  work. 
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Seriously,...  it's  because  for  people/projects  to  be 
successful,  high  maturity  behavior  needs  to  be 
institutionalized,  i.e.,  a  part  of  your  day-to-day  life. 


NORTHROP  GRUMMAN 


OPM:  On  Schedule  to  Achieve  Business  Objectives 


High  maturity  is  continuing  to  help  us  meet  our  “business  objectives”. 
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Summary 


NORTHROP  GRUMMAN 


•  High  maturity  process  areas  are  not  isolated  tunnels. 

•  High  maturity  organizations  continually  use  all  4  process  areas.  They 
do  not  say,  "QPM  and  CAR"  are  not  mine,...  that's  only  for  projects,  or 
"OPP  and  OPM"  are  not  mine,  that's  only  for  organizations.  They  see 
value  in  using  all  4  process  areas. 

•  This  presentation  shows  how  much  you  could  use  high  maturity 
concepts,  even  in  your  personal  lives,  to  be  more  successful. 
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Diane.Mizukami@ngc.com,  310-921-1 939 


The  Issue  of  Performance 

Why 

Do  you  need  a  Maturity  Level  5 

Achieving  the  Organizational  Business 
Objectives  Through  Optimized  Operational 
Processes 


CMMI  ML  4  &  5  PAs  Recap 

Quantitative  Project  Management 
Organizational  Process  Performance 
Causal  Analysis  and  Resolution 
Organizational  Innovation  and  Deployment 


Specific  Practices  of  QPM 


SG  1  Quantitatively  Manage  the  Project 

SP  1 . 1  Establish  the  Proj  ect’  s  Obj  ectives 

SP  1.2  Compose  the  Defined  Process 

SP  1.3  Select  the  Subprocesses  That  Will  Be  Statistically  Managed 
SP  1.4  Manage  Project  Performance 
SG  2  Statistically  Manage  Subprocess  Performance 

SP  2. 1  Select  Measures  and  Analytic  Techniques 
SP  2.2  Apply  Statistical  Methods  to  Understand  Variation 
SP  2.3  Monitor  Performance  of  the  Selected  Subprocesses 
SP  2.4  Record  Statistical  Management  Data 


Project’s  Quality  and 
Process-Performance 
Objectives 


Establish 

Project’s 

Objectives 


Compose 
the  Defined 
Process 


Remedial 

Actions 


f  Select  the  ^ 
Subprocesses 
that  Will  Be 
Statistically 
S.  Managed  y 


Manage 

Project 

Performance 


Statistically  Manage  Subprocess  Performance 


f  Apply 
Statistical 
Methods  to 
Understand 
Variation 


/  Monitor  \ 
f  Performance  ' 
of  the  Selected 
i  Sub- 
V  processes  ) 


Record 

Statistical 

Management 

Data 


uantitati  ve  Proj  ect  Management 

Context 


Quantitatively  Manage  the  Project 


IPM 


Selected 

Subprocesses 


Predictions  of 
Quality  and 
Process 
Performance 


Organization  s 
Measurement 
Repository 


Subprocess 

Capability 


QPM  Summary 


j  QPM  involves  both  quantitative  and  statistical  management.  The  project 

•  establishes  quantitative  objectives  based  on  the  organization’s 
business  objectives  and  needs  of  the  customer 

•  composes  a  defined  process  based  on  historical  capability  data  that 
will  help  it  meet  those  objectives 

•  monitors  the  project  quantitatively  to  assess  whether  the  project  is  on 
course  to  achieve  its  objectives. 

For  each  subprocess  to  be  statistically  managed, 

•  objectives  are  established  for  its  process  performance 

•  its  variation  is  understood  (subprocess  is  stable) 

•  when  the  subprocess  fails  to  achieve  its  objectives,  corrective  action  is 
taken 


Specific  Practices  of  OPP 


SG  1  Establish  Performance  Baselines  and  Models 
SP  1 . 1  Select  Processes 

SP  1 .2  Establish  Process-Performance  Measures 

SP  1.3  Establish  Quality  and  Process-Performance 
Objectives 

SP  1 .4  Establish  Process-Performance  Baselines 
SP  1 .5  Establish  Process-Performance  Models 


Establish  Performance  Baselines  and  Models 


Establish  > 
Process- 
Performance 
Measures  , 


Selected  Subprocesses  from 
Org.  Std.  Processes 


Establish  > 
Process- 
Performance 
Baselines  > 


Organization’s  Set  of 
Standard  Processes 


Establish  ' 
Process- 
Performance 
Models  j 


Organizational  Process- 
Performance  Baselines 


f  Establish^ 
Quality  and 
Process- 
Performance 
Objectives  y 


Organization’s  Quality 
and  Process-Performance 
Objectives 


Process- 
Performance 
Models  * 


Organizational  Process 
Performance  Context 


MA 


QPM 


The  first  three  SPs  establish  processes  (subprocesses), 
measures,  and  objectives  at  the  organization  level  that  focus 
and  align  the  quantitative  management  activities  of  projects 
(QPM)  with  the  business  objectives  of  the  organization. 

last  two  SPs  take  the  actual  results  obtained  from 
projects  to  create  baselines  and  models  that  enable  the  next 
project  to  predict  what  performance  to  expect  from  selecting 
certain  subprocesses  for  its  use,  and  thereby  assess  its  ability 
to  meet  its  objectives. 


SG  1 


SG  2 


Specific  Practices  of  CAR 

Determine  Causes  of  Defects 
SP  1 . 1  Select  Defect  Data  for  Analysis 
SP  1 .2  Analyze  Causes 
Address  Causes  of  Defects 
SP  2.1  Implement  the  Action  Proposals 
SP  2.2  Evaluate  the  Effect  of  Changes 
SP  3.2  Record  Data 


m 

:vA.. 


WM 


ausal  Analysis  and  Resolution 
ns  Context 


Determine 
I  Causes  of  Defects 


4i*0 


Analyze 

Causes 


Action  Proposals 


Select 
Defect  Data1 
for 

Analysis 


Implemenr 
the  Action 
Proposals, 


Record 

Data 


Address  Causes 
of  Defects 


Evaluate 
the  Effect 
of  Change; 


Selected  Action 
Proposals  & 
Improvement 
Proposals 


Performance 

Measures 


a 


o 


QPM 


Defects  and 
Problems 


CAR  Records 


CAR  has  its  greatest  value  when  performed  in  the 
context  of  a  quantitatively  managed  process. 

CAR  involves 

•  a  selection  of  defects  or  problems  whose  resolution 
would  benefit  the  organization 

•  a  root  cause  analysis 

•  development  and  implementation  of  an  action  plan  to 
remove  the  root  causes  of  the  defects  or  problems 


Specific  Practices  of  OID 


SG  1  Select  Improvements 

SP  1 . 1  Collect  and  Analyze  Improvement  Proposals 
SP  1 .2  Identify  and  Analyze  Innovations 
SP  1.3  Pilot  Improvements 
SP  1 .4  Select  Improvements  for  Deployment 
SG  2  Deploy  Improvements 

SP2.1  Plan  the  Deployment 
SP  2.2  Manage  the  Deployment 
SP  2.3  Measure  Improvement  Effects 


f  Collect  \ 
and  Analyze 
Improvement 
Proposals 


Identify 
and  Analyze 
Innovations 


Pilot  Evaluation  Reports 
Lessons  Learned 


Candidate  Innovative 
Improvements 


Process  and  Technology 
Improvement  Proposals 


Updated  Training 
Materials 


eploy  Improvements 


Deployment  Plan 


Measurement 

Results 


Measure 

Improvement 

Effects 


Plan  the 
Deployment 


Manage  the 
Deployment 


Updated  Processes 
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Organizational  Innovation  and 
Deployment  Context 


!Ct  Improvements 


Pilot 
Improvements! 


Select 

Improvements] 
for 

Deployment, 


OID  uses  the  quantitative  information  developed  at  ML4  to 
identify,  analyze,  and  select  incremental  and  innovative 
rovements  to  the  organization’s  processes  and 
echnologies. 

OID  involves  both  incremental  improvement  (everyone  in 
the  organization  is  involved)  and  revolutionary  improvements 
(outward  looking  and  opportunistic)  to  targeted  processes. 

Improvements  are  introduced  systematically  in  the 
organization  by  conducting  pilots,  analyzing  costs  and 
benefits,  and  planning  and  managing  deployment. 

OID  embodies  continuous  improvement  that  results  from 
implementing  all  the  PAs  in  the  model. 


The  Theory  in  the  Model  is  Nice 


However 

Real  Life  is  More  Complicated 

Much  More 


onsiderations  for  Optimization 

Optimization  is  successful  when  the  cost  of 
manufacturing  will  drop  and  your  profit  will 
increase 

Produce  high-quality  products  within  shorter  time 
lines 

To  Correct  balance  between  time  and  cost  versus 
yield  and  quality  is  essential  to  maximize  return 
on  investment 


onsiderations  for  Optimization 


Demonstration  of  the  scalability 

Partial  selection  of  what  to  optimize 

•  Material 

•  Cost  of  product 

•  Design  for 

Scalability 
Availability 
Reliability 
Serviceability 
Maintainability 
Supportability 
Stability 
Reusability 

•  Sustainability  of  the  Technology  as  a  solution 


Main  Steps  for  High  Maturity 
Process  Improvement 


During  our  analysis  and  planning,  we  were  able  to  identify 
improvement  targets  in  main  lifecycle  areas  such  as 

•  operations, 
information, 

•  governance, 
people 

organizational  structure, 
portfolios, 

•  project  execution, 
finance. 

And  as  in  core  process  that  are  critical  to  the  system  success  such 
stakeholder  management,  technical  interfaces  and  integration. 
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Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System 
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Main  Steps  for  High  Maturity  Process 

Improvement 


s  the  result  of  this  observation  we  have  built  an  action  plan, 

Then  in  the  second  step  we  have  built  a  interfaces  map 

using  the  lifecycle  model,  which  enable  us  to  begin  the 
improvement  journey,  and  manage  the  transformation  to 
higher  maturity  by  building  on  each  successive  step,  and 
ultimately  delivering  the  benefits  expected: 

•  reuse 

•  improved  compatibility  and  quality 

•  response  time 

•  interoperability 

•  business  agility. 

Process  performance  and  its  impact  on  the  organization 
governance  is  a  significant  part  of  that  journey 


Case  Studies 

•  Process  Optimization  (Brief  Walkthrough) 

•  Product  Optimization  (Brief  Walkthrough) 

•  Product  Optimization  Which  Leads  to  Process 
Optimization  (Detailed  Walkthrough) 


Suggested  Measures 

Planned  and  Managed  Process 


Availability  and 
completeness  of  plan 

Plan  for  resource 

Overall  performing 
time 

Omissions  in 
performance 

Compliance  to  plan 
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Process  Levels  and  Dimensions 

I  Architected  and  Improved  Process 


Objectives 

Structured 

i 

Monitored  /  Measured 

Effective  /  Efficient 

Process  Interfaces  and 
Integration  in 
Lifecycle 

Prioritize  and  Balance 
Resource  Utilization 
within  Larger  Context 
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Suggested  Measures 

Architected  and  Improved  Process 


i-  Process  productivity 

III  Process  resources 
Bfc,  utilization  effectiveness 

|p  Process  resources 
'  utilization  efficiency 

Meeting  the  process 


objectives 

Other  processes  interfaces 
efficiency 

Process  related  defects 
density 


Process  Levels  and  Dimensions 

► 

j  Operationally  Optimized  Process 


Known  Capability  and  Stable 

Defined  Ingredients 

Known  Critical  Elements 

Meeting  Objectives 

Controlled  Interfaces 

Responsive  /  Modifiable 

Resilience  /  “Agile” 

Relevant  ‘What  If  s  Scenarios 

Accepted  Tolerance  /  Freedom 
Boundaries 

Predictable  Outcomes 


Suggested  Measures 

Operationally  Optimized  Process 


Influence  of  Critical  Elements 
on  process  output 

Process  resources  utilization 
‘What  If  s  Scenarios 

Process  elements  capability 

Quantitative  definition  of 
process  ingredients 


Product  Levels  and  Dimensions 


Planned  and  Managed  System 

Architected  and  Engineered  System 

Operationally  Operated  and  Optimized 
System 


Suggested  Measures 

Planned  and  Managed  System 


|  Requirements  Status 
i  Change  Request  Status 
f  Component  Status 

Increment  Content  -  Components 
Increment  Content  -  Functions 
Technical  Performance 

(Standards  Compliance 
Requests  for  Support 
Support  Time  Requirements 


■Xr>. 


Product  Levels  and  Dimensions 

t- 

Architected  and  Engineered  System 


Operational  Needs  and 
5  Scenarios 

?  System  Architecture 

System  Interfaces  and 
Integration 

P  Validity  /  Verifiability 

Compliance  with 
CONOPS 


Suggested  Measures 

Architected  and  Engineered  System 


|  Maintenance  Actions 
1  Technical  Performance 
f  Performance  Rating 
Requirements  Coverage 
Defect  Containment 
Utilization 

I  Reuse  level 

Interfaces  performance 
Validation  accuracy 


Product  Levels  and  Dimensions 

I  Operationally  Optimized  System 


Scalability 

Availability 

Reliability 

Serviceability 

Maintainability 

Supportability 

Stability 

Reusability 

Soundness  of 
Technology  Future 
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Suggested  Measures 

Operationally  Optimized  System 


Technology  flexibility 

I  Capacity  growth  models 

|  System  (size)  growth 
\  models 

Time  to  Restore 

i 

I  Down  time 
MTBF 

Support  calls  causes  and 
density 

Technology  extendibility 


Case  studies 
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Planning  Data  Defect  Data  Defect  Seepage  XmR  Chart  for  Effort  Variation 
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IT2  IT  T est  Case  Review 
IT  3  Integration  T esting 
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Planning  Data  Defect  Data  Defect  Seepage  XmR  Chart  for  Effort  Variation  XmR  Chart 
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Planning  Data  Defect  Data  Defect  Seepage  XmR  Chart  for  Effort  Variation  •  XmR  Chart 
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Document  Review  Defects 
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Today’s  Talk 


Scope  of  the  presentation 


More  results  from  the  2009  high  maturity  survey  in  the  SEI’s  series  of 
measurement  &  analysis  state-of-the-practice  sample  surveys 


•  Surveys  of  appraised  organizations  in  2008  &  high  maturity  lead  appraisers 
(HMLAs)  in  2009 

—  Great  deal  of  consistency  in  replies  &  statistical  relationships 
—  Based  on  perspectives  of  two  groups  that  are  often  thought  to  be  quite 
different 


•  Focus  here  today  on  results  linking  process  performance  modeling  to 
success  in  achieving  organizations’  appraised  high  maturity  level  goals 


Summary,  lessons  learned  &  next  steps 
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Why  is  This  Work  Important? 


Confusion  still  exists  about  what  is  necessary  to  meet  the  high  maturity 
goals  of  CMMI  based  process  improvement 

•  &  the  value  of  improving  measurement  capability  often  isn’t  appreciated  in 
lower  maturity  organizations 

We  need  more  &  better  measured  evidence  about  the  quality  & 
performance  outcomes  that  are  possible  with  CMMI-based  process 
improvement 

•  Especially  in  these  times  of  limited  resources  &  increasing  skepticism 

•  Without  such  evidence  continued  support  for  CMMI  is  at  increasing  risk 
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The  Wider  Body  of  Work 


In  depth  studies 

•  SEI’s  series  of  workshops  on  measurement  &  analysis  activities  in  high 
maturity  organizations 

—  In-depth  presentations  by  leaders  in  the  field:  5  workshops  over  3  years 
—  Focused  initially  on  value  added  by  CMMI-based  process  performance 
modeling 

—  Detailed  discussions  of  modeling  methods  &  results 
—  Additional  collaboration  within  &  across  organizations 
—  (References  &  URLs  are  at  the  end  of  this  presentation) 

•  Similar  work  ongoing  through  SEI  Partner  Network  working  group  of  leading 
HMLAs 

Wider  publication  of  results 
Additional  workshops  &  colloquia  (TBD) 

Other  ideas? 

•  Let’s  talk  later! 
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Sample  Surveys 


Measurement  &  analysis  state-of-the-practice  series  since  2006 

•  &  occasionally  since  1 990 

•  A  basis  for  more  broadly  based  comparisons  across  programs  &  organizations 

•  Focus  on  value  added  by  CMMI  process  performance  modeling  since  2007 

—  As  it  varies  with  the  extent  of  use  &  understanding  of  PPMs 
—  Along  with  other  aspects  of  measurement  &  analytical  practices 
—  As  well  as  organizational  resources  &  management  support 
—  2009  response  rate:  55% 

A  little  about  validity  &  generalizability  of  sample  surveys... 

•  Candid  replies  recognizing: 

...  (Particularly  seen  in  open-ended  text  responses)  ... 

—  Weak  points  &  need  for  improvement 
—  Backsliding  over  time 
—  Failure  to  achieve  appraisal  goals 

•  Consistent  covariation  with  performance  outcomes 

—  Business  value 
—  Achieved  maturity  levels 
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Today’s  Talk 


Scope  of  the  presentation 


More  results  from  the  2009  high  maturity  survey  in  the  SEI’s  series  of 
measurement  &  analysis  state-of-the-practice  sample  surveys 


•  Surveys  of  appraised  organizations  in  2008  &  high  maturity  lead  appraisers 
(HMLAs)  in  2009 

—  Great  deal  of  consistency  in  replies  &  statistical  relationships 
—  Based  on  perspectives  of  two  groups  that  are  often  thought  to  be  quite 
different 


•  Focus  here  today  on  results  linking  process  performance  modeling  to 
success  in  achieving  organizations’  appraised  high  maturity  level  goals 


Summary,  lessons  learned  &  next  steps 
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How  do  the  Samples  differ? 


To  review  from  last  year... 

2008:  Sponsors  of  organizations  appraised  at  maturity  level  4  &  5 

2009:  HMLAs  asked  to  answer  from  perspective  of  organizational  units 

•  With  which  they  worked  &  were  most  recently  appraised  for  CMMI-based 
high  maturity  status 

•  As  lead  appraisers,  appraisal  team  members,  or  in  a  coaching  capacity 

Still  a  relatively  small  number  of  HMLAs  with  direct  experience  at  time  of 

2009  survey 

•  Realized  sample  would  include  HMLA  replies  about  organizations  that  did 
not  achieve  appraised  high  maturity  goals 

•  Useful  to  better  understand  HMLA  reports  about  use  &  value  added  by 
analytical  approaches  &  methods  used  for  process  performance  modeling 
among  organizations  seeking  appraised  high  maturity  status 
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HM  LA  Role 


Lead  appraiser 
H  Appraisal  team  member 
High  maturity  practices  coach 


n=83 
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The  Survey  Data  Do  Not  Speak  for  Themselves 


Perceptions  &  expectations  often  differ  among  survey  respondents 

•  &  they  probably  do  by  maturity  level 

•  e.g.,  standards  about  how  much  is  enough  to  justify  a  positive  answer 

We’re  not  claiming  cause  &  effect 

•  It’s  statistical  association  at  one  point  in  time 

•  Cause  &  effect  often  are  recursively  reciprocal  over  time 

•  Keep  this  in  mind  as  we  review  the  evidence! 

Results  described  more  fully  in  two  SEI  technical  reports 

•  CMU/SEI-2008-TR-024 

•  CMU/SEI-201 0-TR-022 

•  (References  &  URLs  are  at  the  end  of  this  presentation) 
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Comparable  Organization  Scope  in  Both  Years 


Similar  organizational  context  in  the  2008  and  2009  high  maturity  surveys 

•  Sector  (commercial,  contracted  new  development,  in-house  or  proprietary 
development  or  maintenance,  defense  contractors,  other  government 
contractors,  DoD  or  military  organizations) 

•  Focus  (product  or  system  development,  maintenance  or  sustainment, 
acquisition,  service  provision) 

•  Engineering  discipline  (software,  systems,  hardware,  design,  test) 

•  Number  of  FTE  software,  hardware  or  systems  engineering  employees 

They  do  differ  somewhat  by  country 

•  More  from  China  &  relatively  fewer  from  India 

•  Possibly  since  the  2009  engagements  are  more  recent 
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Synopsis  &  Implications 


As  noted  last  year:  Overall  results  from  HMLAs  in  2009  are  consistent  with 
reports  from  HM  appraisal  sponsors  in  the  2008  survey 

Considerable  understanding  &  use  of  process  performance  models 
(PPMs)  is  evident  in  both  surveys 

Judgments  about  value  added  by  process  performance  modeling  also  vary 
predictably  along  with  variation  in: 

•  Understanding  &  reported  use  of  process  performance  models 

•  Use  of  various  analytical  methods  &  management  processes 

Whether  or  not  the  organizations  achieved  their  appraised  high 
maturity  goals  also  varies  predictably  for  the  same  reasons 

We  hope  to  see  how  &  if  high  maturity  organization  sponsor  perspectives 
continue  to  mirror  those  of  HMLAs  in  future  surveys 

•  With  respect  to  increasing  use  of  analytical  techniques  to  inform 
decision-making 


_  Performance  Effects  of  Measurement  and  Analysis 

Software  Engineering  Institute  CamegieMellon  201 0  ' 

©2010  Carnegie  Mellon  University 


2009:  Statements  about  the  possible  results  of  using  process 

performance  modeling 
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Overall  Value  of  PPMs 


In  your  best  judgment,  how  useful  have 
process  performance  models  been  for  this 
organization  overall? 


U  Extremely 
Valuable 

H  Very 
Valuable 

Bi  Mixed 
Value 

H  Little/No 
Value 


n-75 


Extremely  valuable: 

they  rely  on  them  extensively 

Very  valuable: 

they  have  obtained  much 

useful  information  from  them 

Mixed  value: 

they  have  obtained  useful 
information  on  occasion 

Little  or  no  value 

It's  been  harmful,  not  helpful 
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Achieved  Maturity  Level  Varies  Too 


U  CM  Ml  Maturity  Level  3 
or  lower 

H  Close  To  Maturity  Level  4 
J  CM  Ml  Maturity  Level  4 
LI  CM  Ml  Maturity  Level  5 


LI  Don't  know 


n=82 
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The  Two  are  Closely  Related 


70% 
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Littte/No  Value  Mixed  value  Very  valuable 


y  =  .7Qt  n  =  65 


■  Did  Not 
Achieve 
ML  Goal 
<n=21) 

M  Achieved 
ML  Goal 
(n=44) 


Extremely 

valuable 
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This  may  be  the  most  important  graphic  in  the 
whole  presentation.  Why? 
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What  causes  what? 


Cause  &  effect  are  reciprocal  over  time. 
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The  Chicken  or  the  Egg? 


Cause  &  effect  may  be  reciprocal  over  time 

•  But  it  doesn’t  matter  which  comes  first! 

Organizations  that  find  value  in  their  PPMs  are  more  likely  to  achieve  their 
HM  goals 

But  achieving  HM  status  also  affects  the  likelihood  that  organizations  will 
find  additional  value  in  creating  &  using  an  enhanced  suite  of  PPMs 

•  Achieving  HM  status  also  affects  the  use  of  related  analytical  methods  & 
model  results  to  inform  business  and  technical  decision  making 

Some  organizations  undoubtedly  have  implemented  processes  that  are 
consonant  with  CMMI  best  practices  for  other  reasons 

•  Especially  early  adopters  of  CMMI 

But  the  fact  of  the  matter  is  that  they  are  CMMI  best  practices. 
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Selected  Relationships 


Not  surprisingly:  Strength  of  relationships  with  a  wide  variety  of  factors  are 
often  similar  for  both: 

•  Overall  value  attributed  by  HMLAs  to  use  of  process  performance  modeling 

•  Whether  or  not  the  organizations  achieved  their  appraised  high  maturity 
level  goals 

They  both  vary  predictably  with: 

•  Understanding  &  reported  use  of  process  performance  models 

•  Use  of  various  analytical  methods  &  management  processes 

See  the  TR  &  last  year’s  presentation  for  more  detail  about  overall  value... 

•  (References  &  URLs  are  at  the  end  of  this  presentation) 

What  follows  here? 

•  The  strongest  relationships  with  achievement  of  high  maturity  level  goals 

•  Highlighting  instances  where  strength  of  those  relationships  differ  from 
comparable  relationships  with  overall  value 
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Healthy  PPM  Ingredients:  Emphasis 


How  much  emphasis  does  the  organization  place  upon  the  following  in  its 
process  performance  modeling? 

•  Accounting  for  uncertainty  and  variability  in  predictive  factors  and  predicted 
outcomes 

•  Factors  that  are  under  management  or  technical  control 

•  Other  product,  contractual  or  organizational  characteristics,  resources  or 
constraints 

•  Segmenting  or  otherwise  accounting  for  uncontrollable  factors 

•  Factors  that  are  tied  to  detailed  subprocesses 

•  Factors  that  are  tied  to  larger,  more  broadly  defined  organizational 
processes 

Note  that  values  on  the  extremes  of  this  &  all  other  weighted  sum 
measures  require  consistency  of  replies  across  all  of  the  component 
sub  questions 
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Survey  Composite  Variables 


Many  of  the  relationships  described  here  use  composite  measures  that  are  based  on  combinations  of 
several  related  component  questions.  Each  composite  measures  is  based  on  the  appraisers’  answers  to  a 
group  of  related  questions.  The  possible  answers  to  those  questions  are  ordinal  categories  such  as 
extensive,  substantial,  moderate,  limited  &  little  if  any. 

Some  of  the  composite  measures  are  simple  counts.  Others  such  as  this  one  are  weighted,  summed  index 
of  the  respondents’  answers  to  each  of  the  questions.  Much  like  a  grade-point  average,  the  answers  are 
assigned  ordered  numeric  values  that  are  added  and  then  divided  by  the  number  of  valid  answers  to  the 
series  of  questions  for  each  respondent.  For  example  extensive  answers  are  scored  as  the  value  5, 
substantial  as  4,  down  to  little  if  any  as  1 .  Hence  the  values  on  the  extremes  of  the  weighted  sum 
measures  require  consistency  of  replies  across  all  of  their  respective  component  questions.  The  index 
scores  are  separated  into  categories  based  on  the  distribution  of  the  total  response  values  for  ease  of 
interpretation. 

The  weighting  &  summing  are  mathematically  equivalent  to  an  arithmetic  mean;  however,  also  much  like  a 
grade  point  average,  the  results  are  rank  orders.  Such  indices  are  not  interval-  or  ratio-level  measures  that 
can  be  added  or  multiplied  meaningfully. 

See  Appendix  C  in  CMU/SEI-2010-TR-022  (http://www.sei.cmu.edu/librarv/abstracts/reports/10tr022.cfm) 
for  further  detail. 
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Relationship  Between  Healthy  PPM  Ingredients  & 
Overall  Value  Attributed  to  PPMs:  Emphasis 


ingredients:  Emphasis 


Still  room  for 
improvement  in 
PPM  emphasis 

Which  does  seem  to 
pay  off 


Possible 

answers: 

•  Extensive 

•  Substantial 

•  Moderate 

•  Limited 

•  Little  if  any 
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Interpreting  the  Mosaics  &  Gamma 


Most  of  the  results  described  here  summarize  relationships  between  two  variables.  Many  of  them  use  a 
graphical  mosaic  such  as  this  one  to  shows  the  extent  to  which  the  survey  respondents’  answers  vary 
together  in  a  consistent  manner. 

The  values  for  each  x-variable  are  displayed  along  the  horizontal  axis  on  the  bottom  of  the  mosaic,  and 
labels  for  the  respondents’  answers  to  the  y-factor  are  displayed  to  the  right  of  the  mosaic  on  the  vertical 
axis.  The  proportions  of  responses  for  each  category  of  the  x-variable  are  shown  in  separate  columns  of 
the  mosaic,  where  each  value  of  the  y-variable  is  represented  in  a  separate  mosaic  tile.  The  width  of  each 
column  varies  in  proportion  to  the  number  of  responses  for  each  category  of  the  x-variable.  This  can 
provide  a  quick  sense  of  how  evenly  or  unevenly  the  survey  answers  are  distributed. 

The  overall  strength  of  the  relationship  between  the  two  variables  can  be  described  by  the  value  of  the 
gamma  statistic.  Gamma  is  an  ordinal  measure  of  association  that  is  appropriate  for  ordered 
categorical  measures  such  as  these.  It  is  symmetric,  which  means  that  its  value  will  be  the  same 
regardless  of  which  variable  is  considered  to  be  an  x-variable  or  a  y-variable. 

The  value  of  gamma  is  the  proportion  of  paired  comparisons  where  knowing  the  rank  order  of  one  variable 
reduces  the  proportionate  error  in  predicting  the  rank  order  of  the  other  variable.  So,  for  example,  if  gamma 
is  .75  then  knowing  the  independent  variable  reduces  our  error  in  predicting  the  rank  of  the  dependent 
variable  by  75  percent. 

See  Appendix  C  in  CMU/SEI-2010-TR-022  (http://www.sei.cmu.edu/librarv/abstracts/reports/10tr022.cfm) 
for  further  detail. 
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Healthy  PPM  Ingredients  &  Achievement  of 
Appraisal  Maturity  Level  Goal:  Emphasis 


■  Did  Not 
Achieve 
ML  Goal 
(n=21) 

H  Achieved 
ML  Goal 
(n=44) 


^Moderate  Moderate  to  Toward  but 
<  midway  toward  ^substantial 
substantial 


Substantia  tor 

better _ 

y  =  .75,  n  =  65 
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Healthy  PPM  Ingredients:  Purpose 


To  what  degree  are  your  organization’s  process  performance  models  used 
for  the  following  purposes? 

•  Predict  final  project  outcomes 

•  Predict  interim  outcomes  during  project  execution  (e.g.,  connecting 
“upstream”  with  “downstream”  activities) 

•  Model  the  variation  of  factors  and  understand  the  predicted  range  or 
variation  of  the  predicted  outcomes 

•  Enable  “what-if”  analysis  for  project  planning,  dynamic  re-planning  and 
problem  resolution  during  project  execution 

•  Enable  projects  to  achieve  mid-course  corrections  to  ensure  project 
success 
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Relationship  Between  Healthy  PPM  Ingredients  & 
Overall  Value  Attributed  to  PPMs:  Purpose 


Healthy  PPM  Ingredients  &  Achievement  of 
Appraisal  Maturity  Level  Goal:  Purpose 


Did  Not 
Achieve 
MLGoal 
n-Z2 

Achieved 
ML  Goal 
n=44 


<Moderate 


Moderate  to 
<midway 
toward 
substantial 


Toward  but 
substantial 


Substantialor 

better 
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Statistical  Analysis  Methods 


To  what  extent  are  the  following  statistical  methods  used  in  the 
organization’s  process  performance  modeling? 

•  Regression  analysis  predicting  continuous  outcomes  (e.g.,  bivariate  or 
multivariate  linear  regression  or  non-linear  regression) 

•  Regression  analysis  predicting  categorical  outcomes  (e.g.,  logistic 
regression  or  loglinear  models) 

•  Analysis  of  variance  (e.g.,  ANOVA,  ANCOVA  or  MANOVA) 

•  Attribute  SPC  charts  (e.g.,  c,  u,  p,  or  np) 

•  Individual  point  SPC  charts  (e.g.,  ImR  or  XmR) 

•  Continuous  SPC  charts  (e.g.,  XbarR  or  XbarS) 

•  Design  of  experiments 
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Relationship  Between  Use  of  Multiple  Statistical 
Methods  &  Overall  Value  Attributed  to  PPMs 
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Use  of  Multiple  Statistical  Methods  & 
Achievement  of  Appraisal  Maturity  Level  Goal 


Use  of  Statistical  Methods  for  Process  Performance 

Models 


B  Did  Not 
Achieve 
ML  Goal 
(n=23) 

■  Achieved 
ML  Goa! 
(n=45) 


^Moderate  Moderate  to  Toward  but 

< midway  toward  < su bst a nt ial 

substantial 


Substantial  or 
better 


y  =  .60t  n  =  68 
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Simulation/  Optimization  Techniques 


Which  of  the  following  other  optimization  approaches  are  used  in  your 
organization’s  process  performance  modeling? 

•  Monte  Carlo  simulation 

•  Discrete  event  simulation  for  process  modeling 

•  Markov  or  Petri-net  models 

•  Probabilistic  modeling 

•  Neural  networks 

•  Optimization 
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Relationship  between  #  of  Simulation/Optimization 
Techniques  used  &  overall  value  attributed  to  PPMs 


#  of  Simulation/Optimization  Techniques  Used  & 
Achievement  of  Appraisal  Maturity  Level  Goal 


Number  of  Simulation/Optimization  Techniques  Used 


50% 

45% 

40% 

35% 

30% 

25% 

20% 

15% 

10% 

5% 

0% 


■  Did  Not 
Achieve 
ML  Goal 
(n=19) 

m  Did 
Achieve 
ML  Goal 
(n=42) 


None  1  2  3 _ 

y  =  .89,  n  =  61 
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Use  of  Specific  Simulation/Optimization 
Techniques  &  Achievement  of  Appraisal  ML  Goal 


80% 


70% 


60% 


50% 


40% 


10% 


20% 


10% 


Q% 


It’s  not  just  what  techniques  you  use. 
It ’s  how  &  why  you  use  them... 


J 


Neural  Optimization 
networks 


Other 


Oid  Not 
Achieve 
ML  Goal 
fn-19) 

i  Achieved 
ML  Goal 
(n=42) 


Y=  .51 


y  =  .80  y=  1  -0  y=  -91 


Y=  .50  y=-05  y=-.87 


_  Performance  Effects  of  Measurement  and  Analysis 

Software  Engineering  Institute  CamegieMellon  201 0  3 

©2010  Carnegie  Mellon  University 


It’s  the  uses  to  which  the  results  are  put... 


Measurement  &  analytical  methods  aren’t  all  that’s  important  for  achieving 
high  maturity  status 


Management  processes  are  important  too! 

•  With  active  participation  of  the  intended  users  of  the  measurement 
results 

•  Who  in  turn  are  well  equipped  to  understand  the  results 
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Relationship  Between  Use  of  PPM  Predictions  in 
Reviews  &  Overall  Value  Attributed  to  PPMs 


Extremely  Valuable 


Very  Valuable 


Mixed  Value 


Little  Value 


o 

_a 

< 


Use  of 
PPMs 


How  often  are 
process 
performance 
model  predictions 
used  to  inform 
decision  making 
in  the 

organization’s 
status  and 
milestone 
reviews? 


Of  interest  as  a 
performance 
measure  in  its  own 
right 

Also  for  its  impact 
on  overall  outcome 
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Use  of  PPM  Predictions  in  Status/Milestone 
Reviews  &  Achievement  of  Appraisal  ML  Goal 


100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


H  Did  Not 
Achieve 
ML  Goal 
(n=21) 


Achieved 
ML  Goal 
(n-41J 


Rarely  Occasionally  About  half  Frequently 
if  ever  the  time 


Almost 

always 


7  =  .62,  n=62 
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Quality  of  PPM  Documentation  &  Overall  Value 
Attributed  to  PPMs 


How  would  you 
best  describe 
the 

organization's 
documentation 
of  its  process 
performance 
and  quality 
measurement 
results  (e.g.,  in 
the 

organization's 
PI  IDs, 

measurement 
repository  or 
process  asset 
library)? 
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Quality  of  PPM  Documentation  &  Achievement  of 
Maturity  Level  Appraisal  Goal 


50% 

45% 

40% 

35% 

30% 

25% 

20% 

15% 

10% 

5% 

0% 


■  Did  Not 
Achieve 
ML  Coal 
In-23) 

u  Achieved 
ML  Goal 
fn-44) 


Fair 


Ad  equate 


Good 


Excellent 


y  =  .86,  n=67 
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Relationship  Between  Managers’  Understanding  of 
Model  Results  &  Overall  Value  Attributed  to  PPMs 


How  well  do 
the 

managers  in 
the 

organization 
who  use 
process 
performance 
model  results 
understand 
the  results 
that  they 
use? 
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Relationship  Between  Managers’  Understanding  of 
Model  Results  &  Achievement  of  ML  Goal 


45% 


■  Did  Not 
Achieve  ML 
Goal  n=23 


M  Achieved  ML 
Goal  n~47 


Hardly  at  all  To  some 

extent 


Moderately  Very  we  1 1 
well 


Managers'  Understanding  of  PPM  Results 


Extremely 
well 


y  =  .78,  n  =  70 


_  Performance  Effects  of  Measurement  and  Analysis 

Software  Engineering  Institute  CamegieMellon  201 0  1 

©2010  Carnegie  Mellon  University 


Stakeholder  Involvement 


How  would  you  characterize  the  involvement  of  various  potential 
stakeholders  in  setting  goals  and  deciding  on  plans  of  action  for 
measurement  and  analysis  in  your  organization? 

•  Customers 

•  Executive  and  senior  managers 

•  Middle  managers  (e.g.,  program  or  product  line) 

•  Project  managers 

•  Project  engineers  and  other  technical  staff 

•  Process  and  quality  engineers 

•  Measurement  specialists 


As  per  GQ(I)M 

Measurement  & 
Analysis  SGI,  SP1 

As  well  as  GP  2.7 
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Relationship  Between  Stakeholder  Involvement  & 
Overall  Value  Attributed  to  PPMs 


Stakeholder 
i  n  vo  I  ve  ment 
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Relationship  Between  Stakeholder  Involvement  & 
Achievement  of  ML  Goal 


4596 

4096 

359& 

3096 

2596 

2096 

1596 

1096 

596 

096 


Did  Not 
Achieve 
ML  Goal 
n=24 


B  Achieved 
ML  Goal 
n=44 


< 


Moderate 


Moderate  to  < 
rn  i  d  way  to  wa  rd 
substantial 


Midway  toward 
substantial  to  < 
substantial 


Substantial  or 

better  _ 

y  =  .66,  n  =  68 


Stakeholder  Involvement  with  Measurement  Sc  Analysis 
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Relation  Between  Quality  of  Project  Manager 
Training  &  Overall  Value  Attributed  to  PPMs 


How  would  you 
best  characterize 
the  measurement 
related  training 
that  is  available 
(for  project 
managers)  in  the 
organization? 


_  Performance  Effects  of  Measurement  and  Analysis 

Software  Engineering  Institute  CamegieMellon  201 0  1 

©2010  Carnegie  Mellon  University 


Relation  Between  Quality  of  Project  Manager 
Training  &  Achievement  of  ML  Goal 


60% 


50% 

40% 

30% 

20% 

10% 

0% 


H  Did  Not 
Achieve  ML 
Goal  n=22 

H  Achieved 
ML  Goal 
n=48 


Poor  Fair  Adequate  Good  Excellent 


Project  Managers'  Measurement  Training 


y=  .61,  n  =  70 


_  Performance  Effects  of  Measurement  and  Analysis 

Software  Engineering  Institute  CamegieMellon  201 0  1 

©2010  Carnegie  Mellon  University 


Today’s  Talk 


Scope  of  the  presentation 

More  results  from  the  2009  high  maturity  survey  in  the  SEI’s  series  of 
measurement  &  analysis  state-of-the-practice  sample  surveys 


Summary,  lessons  learned  &  next  steps 
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Summary  of  Results1 


Overall  value  of  process  performance  modeling  &  related  activities  was 
judged  to  be  more  valuable  for  organizations  that: 

•  Understood  &  used  measurement  &  analysis  activities  more 
frequently 

•  Provided  organizational  resources  &  management  support  for 
the  work 

Organizations  that  achieved  their  appraised  high  maturity  goals  shared  the 
same  characteristics 

•  Moreover  achievement  of  appraised  high  maturity  status  is 
closely  associated  with  better  project/program  performance, 
product  quality  &  organizational  decision  making 

Replies  to  both  surveys  were  generally  consistent  even  though  the  two 
groups  are  often  thought  to  be  quite  different 
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Summary  of  Results2 


HMLAs  were  slightly  less  positive  overall  in  judgments  of  value  added 

But  somewhat  more  positive  than  the  sponsors  about  consistent  use  of 
process  performance  modeling  approaches  &  analytical  methods 

Some  conjectures  about  why  that  is  so 

•  The  HMLAs  were  reporting  about  more  recent  appraisals  &  coaching 
engagements 

•  The  HMLAs  were  basing  their  judgments  on  evidence  gathered  at  the 
project  or  program  level 

•  The  sponsors  have  a  better  understanding  about  overall  goals  &  objectives 

—  Which  may  not  be  addressed  by  the  process  performance  modeling 
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The  Bottom  Line  Again... 

Responses  to  2009  survey  of  high  maturity  lead  appraisers  are  consistent 
with  the  responses  from  representatives  from  appraised  high  maturity 
organizations  surveyed  in  2008 

The  community  can  be  confident  that  the  appraisers’ judgments  are 
consistent  with  the  organizations’  own  views  of  the  value  of  measurement 
&  analysis  to  their  work 
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Thank  You  for  Your  Attention! 


Jim  McCurley  &  Dennis  Goldenson 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213-3890 
USA 
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Why  do  the  2008  &  2009  High  Maturity  Samples 
Differ? 


Surveyed  high  maturity  lead  appraisers  (HMLAs)  in  2009 

•  Provides  a  useful  comparison  with  those  of  the  perspectives  from  the 
appraised  organizations 

Intent  has  been  to  reuse  &  modify  the  2008-2009  questionnaire  in  future 
years 

•  Using  the  2008  results  as  the  baseline  for  tracking  changes  in  high 
maturity  organizations  over  time 

Only  a  limited  number  of  organizations  have  achieved  high  maturity  status 

But  we  won’t  ask  the  same  people  to  answer  the  same  questions  over  & 
over  each  year 

•  (Surveyed  appraisal  sponsors  &  their  designees  in  2008) 
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Location 


y  United  States 
H  China 
1  India 
M Japan 
1  Al  l  Others 


n=82 
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Type  of  organization 


M  Commercial  OTS 

■  Contracted  new 
development 

H  In-house 

H  Defense  contractor 

El  Other  government 
contractor 

H  DoDefense/military 
organization 

El  Other  government 
agency 

El  Other 
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Primary  focus  of  work 


y  Product  or  system 
development 

B  Maintenance  or 
sustainment 

B  Acquisition 
B  Service  provision 
y  Other 


n=83 
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Which  of  the  following  product  quality  and  project  performance  outcomes  were  routinely  predicted 

with  process  performance  models  in  the  organization? 


Delivered 

Costand 

Accuracy  of 

Tyoeor 

Product 

Customer 

Qua  lity  of 

Work 

ROI  of  Other 

None  of  the 

defects 

schedule 

estimates 

severity  of 

quality 

satisfaction 

services 

product  size 

process 

above 

duration 

(e.g.,  cost. 

defects 

attributes 

orovided 

improvement 

schedule. 

(e.g.,  mean 

(e.g.,  IT  ticket 

or  related 

product  size 

time  to 

resolution 

financia 

or  effort) 

failure, 

time) 

performance 

design 
complexity, 
etc. I 


H  2008  Orgs  (n=144) 
y  2009  L.A.  (n=77) 
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Which  of  the  following  ( often  interim)  process  performance  outcomes  are  routinely  predicted 
with  process  performance  models  in  your  organization? 


Estimates  at 

Escaped  defects 

Effectiveness  or  Cost  of  quality 

Requirements 

Practitioner 

Other 

completion  (Le., 

(e.g.,  as 

efficiencyof  and  poor  quality 

volatility  or 

adherence  to 

performed 
periodically 
throughout  the 
project) 

predicted  by 
defect  phase 
containment 
models) 

inspection  or  (e.g.,  rework) 

test  coverage 

growth 

defined 

processes 

Noneof  the 
above 

H  2008  Orgs  {n  144) 
H2009L.A.{n=7G) 
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Which  of  the  following  processes  and  activities  were  routinely  modeled  in  the  organization? 


90% 


Project 

Qual'ty  control 

Software 

Requirements 

Process 

Systems 

Product 

Hardware 

Acquisition  or 

Other 

planning  ard 

processes 

design  and 

engineering 

documents!  on 

engineering 

architecture 

engineering 

suoplie" 

estimation 

coding 

processes 

processes 

processes 

None'jf  the 
above 


H  2008  Orgs  (n=144) 
H  2009  LA.  (n=77) 
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Sampling  Issues 


Lower  than  desired  response  rates 

Not  surprising  in  relatively  long  questionnaires 


Exacerbated  by: 

•  Repeated  contact  of  the  same  individuals  for  business  as  well  as  survey 
purposes 

•  Demands  on  time  from  busy  executives 

Considering  other  sampling  strategies  for  future  surveys 

“State  of  the  practice”  also  can  refer  to  very  different  target  populations 

•  The  SEI  customer  base  ...  the  broader  software  &  systems  engineering 
community  ...  or  those  organizations  that  more  routinely  use  measurement? 

•  Of  course,  the  population  depends  on  the  purposes  of  the  survey 
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ISRAEL  AEROSPACE  INDUSTRIES 


SYSTEMS  MISSILES  &  SPACE  GROUP 

System  Engineering 
Process  Improvement 
using  the  CMMI 
in  Large  Space  Programs 

Sarit  Assaraf,  Revital  Goldberg 

Israel  Aerospace  Industries 


< WSMMI 
I  'V'Vy  I 


10th  Annual  CMMI  Conference  15-19  Nov .  2010  Denver  CO. 


This  document  contains  proprietary  information  of  Israel  Aerospace  Industries  Ltd.  and  may  not  be  reproduced,  copied,  disclosed  or  utilized  in  any  way  in  whole  or  in  part,  without  the  prior  written  consent  of  Israel  Aerospace  Industries  Ltd 


Israel  Aerospace  Industries  (IAI)  c-“ 


9  Largest  industrial  company  in  Israel 

9  Missiles,  Satellites,  UAVs,  Avionics,  Upgrades, 
RADARS,  etc. 

9  Activities  encompassing:  Development, 

Production,  Maintenance  and  Service  of 
Aerospace  Systems 

s*  IAI  divisions  are  certified  for  IS09000  and  AS9100 
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Process  Improvement  -  WHY? 


(((CM  Ml 


“The  Qualify  of  a  jystem 

is  Governed  by 

the  Quality  of  the  Process 

Used  to  Develop  it” 

Watts  Humphrey, 

“Software  Process  Program”  founder  at  the 
Carnegie  Mellon’s  Software  Engineering  Institute 
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IAI  Process  Improvement  Path 


(((CM  Ml 

% 


Starting  activities  for 

Starting  CAI 

project  management 

activities  and  NPI 

process  improvement 

development 

Starting  Software 
Process 
Improvement 
Program  (SPIP) 
based  on  CMM 


Starting  System 

Process 

Improvement 

(SYSPI) 


Starting  Advanced 
System  &  SW 
Technology: 
based  on  CMMI 


SW-CMM 

SE-CMM 

CMMI 

Current  Status 
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CMMI  &  SEPG  Program  -  Group  Level  (i)  S 


Systems  Missiles 
Space  Group 
YEHUD  Campus 
(4  Div.) 


One 

Three 

Operative  Div. 

Business  Div. 

(Cost  Center) 

(Profit  Centers) 

9  The  CMMI  implementation  is  managed  as  a  program 
for  process  improvement  at  the  group  level 

SAPIR  -  Standard  Annual  Process  Improvement  Roadmap 


9  The  management  concept  is  based  on  the  CMMI  ORG 
^  and  Support  PAs 
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_ 

CMMI  &  SEPG  Program  -  Group  Level  (2)^ 


a  Discipline  based  PITs  -  Process  Improvement  Teams 


PM,  SE,  SW,  HW,  QA,  CM,  SAM,  ORG 


o  PIT  members  =  Process  Leaders  and  Project 
Representatives 

9  Each  PIT  conducts  meetings  and  activities,  managed 
by  PIT  Leader1^ 

9  ORG  PIT  comprises  of  all  PITs  LeadersvT 
9  CMMI  Integrated  plan  consisting  of  all  PITs’  plans 
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IAI  Process  Improvement  Strategy 


(((CM  Ml 


IAI 

Corporate 

Level 


IAI  VP  Operations 
Engineering  &  Development 
CMMI  Program  Office 


IAI  Groups 
and  Divisions 


•Sponsorship 

•Coordination 

•Budget 

•Process  Assets  Development 


•Local  Sponsorship 
•Budget  &  Resources 
•Process  Assets  Adaptation  and 
Development 
•Process  Implementation 
•Process  Feedback 
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Process  Definition  and  Tailoring  Method 


(((CM  Ml 


IAI  Corporate  level 

Company  HDQ 

, - - 

A  set  of  SE  standard  processes 
methodologies,  tools,  tailoring 
guidelines  & 
training  material 


Feedback 


At  the  Group  level 

within  the  SE  PIT 


y 


Feedback 
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Space  Systems 


(EGMMI 


Th®  Satellite  Program  =  A  System  of  Systems  Program 


Venus  Observation 
satellites 


Shavit 

launch 

vehicles 


TECSAR 

Synthetic  Aperture 
Radar 


Communication 
satellites 


AMOS  3 


AMOS  1,2 


AMOS  4 

Advanced  comm.  Sat. 
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OFEQ 


EROS  C 

OPTSAT  3000 


Ground 

control 

stations 


IAI  SPACE  Programs  Main  Characteristics 
Program  Internal  Structure  (i) 


CMMI 


9  Decentralized  SE  at  the  group  level  - 
The  SOS  and  the  Systems  SE  are  part  of  the  program  office, 

also  managing  the  Sub-Contractors. 


9  The  Sub  Systems  SE  are  part  of  the  Engineering 
Departments 


Satellite  Program 


SOS  SE 


Program 

Office 


Eng.  _J 
Departments  L 


Ground  Segment  Satellite  Segment  C^Systems  SE 


Sub  Sys.SE 


9  Different  views  and  meanings  of  SE  activities  adequate  to  each 
level  of  the  system:  SOS,  System,  Subsystem. 


9  Large  project  teams  (often,  geographically  dispersed) 
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IAI  SPACE  Programs  Main  Characteristics 
Program  Internal  Structure  (2) 


- \ 

Program 

Management 
s* 


HW  Eng. 

Satellite/ 

Ground 


SW  Eng. 
Satellite/ 
Ground 
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A 

System  Eng. 
Satellite/ 
Ground 


IAI  SPACE  Programs  Main  Characteristics 
Program  Internal  Structure  (3) 


CMMI 

y  l 


9  FORMAL  mutual  commitment  between  the  Program 
office 

and  the  Engineering  Groups  supports  implementation  of 

PP  SP3.3 ,  PMC  SP1.2,  REQM  SP1.2  &  IPM  SP2.2: 

9  The  Program  Office  issues  Internal  Customer-Supplier 
Agreements 

9  The  Program  Office  allocates  budget  for  the  engineering 
groups  against  their  commitment  to  supply  adequate  products 
on  time 

9  The  program  establishes  Integrated  teams  consisting  of 
different  disciplines  (IPT  =  Integrated  Product  Teams) 
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IAI  SPACE  Programs  Main  Characteristics 
Many  Sub  Contractors  &  Suppliers 


9  Complex  coordination 

>  Internal  (IAI)  sub  contractors 

>  External  sub  contractors 

>  Suppliers 

Supports  implementation  of 


/^lAI/MBT 
^Rjlme  Contract 


Sub  Int.  Sub  Supplier  Sub  Supplier 
Contractor  Contractor  Contractor 


ipm  SP2.1,  SP1.5;  sam  SPs;  ts  SP2.4;  dar 


9  Complex  System  Integration  aspects  -  Integration 
Strategy  planning  requires  special  attention  (pi  SP1.1 ): 


>  Scheduling 

>  Sub  Contractors  Monitoring 

>  Interfaces  definition  correctness  and  completeness  (TS  SP2.3,  PI 
SP2.1) 

>  Integration  Readiness  Review  (PI  SP3.1) 
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IAI  SPACE  Programs  Main  Characteristics 
The  Satellites  “Nature” 


(((CM Ml 

■  y  y 


9  Each  Satellite  is  “One  of  a  Kind” 

however . . . 


9  Verification  &  Validation  are  limited,  especially  by  the 
ability  to  simulate 
the  environmental  conditions. 

Support  implementation  of  ver  &  val  SPf?1,  SP1.2,  SP1.3 


9  Once  Launched  -  Repair  activities  are  limited  to  SW 
corrections  and  updates. 

Requires  special  attention  to: 


y 


>  Requirements  validation  at  early  development  stages  (RD  SP3.5) 

>  Analysis  of  Failures  found  on  PI,  VER  &  VAL  stages 
(PI,  VER  SP3.2,  VAL  SP2.2) 
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Conclusions  &  Lessons  Learned 


(((CMMI 


9  The  IAI  process  deployment  method  relies  on  a 
Corporate  Level  processes  definition,  followed  by 
Group/Division  Level  tailoring  and 
implementation. 


9  The  products  and  projects  characteristics  require 
special  attention  to  the  RD,  PI,  VER  &  VAL 
process  areas 


9  Using  the  internal  Customer-Supplier 

Agreements  and  IPT’s  ensure 


9  Better  definition  and  management  of  the  commitment 
o  Integrated  team  work  along  the  projects’  life  cycle 
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Thank  you 


(((CM  Ml 


Sarit  Assaraf  -  sassaraf@/a/.  co.  // 


Revital  Goldberg  -  rqoldbera(g).iai. co. ij 
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The  Boeing  Company 


Garv.C.Palosaari@boeinq.com 


310-416-3591 


CMMI®  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
Copyright© 2010  Boeing.  All  rights  reserved. 


Boeing  Space  and  Intelligence  Systems  (S&IS) 


S&IS  is  Boeing’s  center  for  all  intelligence  and  government  / 
commercial  space  systems.  It  is  the  world's  leading 
manufacturer  of  geostationary  satellites,  and  has  the  largest 
dedicated  satellite  factory  in  the  world.  S&IS’  headquarters  is  in 
Seal  Beach,  California. 


CMMI®  Core  Team  located  in  El  Segundo,  Southern  California 


Agenda 


■  Process  Management  Challenges 

■  A  well-designed  Process  Architecture  overcomes  the 
Challenges 

■  What  makes  up  a  Process  Architecture? 

■  Steps  to  establish  and  maintain  a  Process  Architecture 

■  Improving  by  using  a  Process  Architecture 

■  Keys  to  success 
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Process  Management  Challenges 


■Too  many  processes 

■  Redundant,  contradictory 

■Too  many  types 

■Too  many  interfaces 
■No  integrated  views  /  roadmaps 
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A  well-designed  Process  Architecture 
Overcomes  the  challenges 


■  Provides  a  bird’s  eye  view 

-  Graphical  representations  establish  a  framework  to 
help  users  find  processes 


■Provides  insight  into  interrelationships  and 
ordering 

-  Helps  with  process  update  impact  analysis  -  fewer 
unintended  consequences 

-  Helps  with  planning,  process  deployment  analysis 
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What  makes  up  a  Process 


■Components  (policies,  processes,  procedures,  etc.) 
grouped  in 

-  Hierarchy 

-  Contextual  relationships 

■Interfaces  and  interdependencies  among  the 
components 

■Ordering  of  the  components 


Note:  For  additional  process  architecture  information  see  CMMI®  (Ver.  1.2)  -  Glossary  and  Organizational  Process  Definition 
(OPD-SP  1.1)... 
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Authority  References 


Hierarchy 


Processes 


Governance 


What-to-Do 


Required 

. 7 


Procedures 


Templates,  Checklists, 
Standards  etc. 


How-to’s 


*See  [Olson  1994]  for  additional  definition  and  relationship  information 
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Increasing  Detail 


Contextual 


Product  View 


Organizational  View 


Business  Model  View 


Marketing  /  Di^^ 

Engineering 

Suppliers 

Billing  /  Accounting 

Purchasings*. 

Human  Resources 

Facilities  Management 

Corporate  Services 


Information  Technology 


*See  [Olson  2008]  for  addition  architecture  examples 
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Interfaces  and  Interdependencies 
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Ordering 


Order  (sequencing,  iteration,  concurrency)  is  captured  through 
process  mappings  to  lifecycles,  value  streams,  or  process  flows... 


Lifecycle  Example 


•  Procedure  Y  •  Procedure  Y 
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Steps  to  Establish  and  Maintain  a  Process 
Architecture* 


1.  Requirements 
Analysis 


2.  Design 


Implementation 


4.  Operations 
&  Maintenance 


Hierarchy 


Contextual  Views 


Interfaces/ 

Interdependencies 


Ordering 


*See  [Chang-Hyun,  2010]  for  additional  information  on  how  to  build  a  software  architecture 
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1.  Requirements  Analysis 


■  Identify  and  analyze  process  architecture  requirements 

-  Use  cases,  scenarios  (finding  /  improving  processes) 

-  Existing  documentation  /  interface  standards 

-  CMMI®,  AS91 00  compliance 

-  Level  of  detail  versus  expert  mode  capability 

-  Weak  versus  strong  associations  between  processes  and  procedures 

-  Process  update  impact  analysis  capability 

■  Identify  constraints 

-  Legacy  processes  /  architectures  (potential  multi-site  issues) 

■  Identify  and  analyze  quality  attributes 

-  Simplicity  /  usability  (processes  should  be  easy  to  find) 

-  Flexibility  and  maintainability  (architecture  should  be  easy  to  update) 

-  Degree  /  levels  of  commonality  (common  processes  can  be  to  generic) 

-  Extensibility,  reusability  (allow  for  future  expansion,  building  blocks) 
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2.  Design 


Analyze  structural  alternatives  and  identify 
solutions 

Identify  layers  of  hierarchy  and  which  components  are  required 

Identify  strength  (level  of  detail)  of  process  mappings 

Establish  process  groupings  and  “look-and-feel”  for  contextual  views 

Identify  process  and  process  element  interfaces 
(e.g.  I/O  data,  givers  /receivers) 

Identify  lifecycle  phases  to  be  process  mapped 
Standardize  process  templates  /  definitions 

Evaluate  tools  /  methods  to  build  /  populate  the 
process  architecture 

Process  Modeling  Tools 
-  Wiki’s 

Process  Flow  Diagrams 


Product  View 
Organizational  View 


Business  Model  View 


Marketing  /  Distribution 

Engineering 

Inventory 

Management 

Suppliers 

Manufacturing 

Billing 

Accou  nl  ing 

F'u  rcihas  ing 

Human 

Resources 

Facilhes 

Management 

Corporate  Services 


Information  Technology 


Note:  See  [Clements  2005]  and  [Clements  2010]  for  additional  information  on  design  principles  and  view  selection  criteria 
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3.  Implementation 


Collect  and  catalogue  components 

-  Map  procedures  (How-to-Do’s)  to  processes 
(What-to-Do’s) 


■Group  processes  and  procedures 
into  contextual  views 


■  Identify  interfaces 


■  Map  to  lifecycles 


)>  Gate  1  Gale  2  >  Gate  3  "p  Gate... 


■  Process  A 

■  Process  A 

-*Ac*U*X 

■  Process  B 

■  5C 

Gate  1A  Gate  2A  > 


1  Process  A  -  Process  A 
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4.  Operations  and  Maintenance 


■  Establish  change  control  mechanisms,  tools,  methods 

■  Monitor  implementation  /  updates  to  ensure  that 
architecture  is  maintained 

■  Integrate  updates  of  your  process  architecture  into  your 
process  management  activities 

■Re-factor  as  necessary,  in  response  to  process  creep, 
acquisitions,  or  expanded  functionality  needs 
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Improving  Operations  with  a  Process 
Architecture 


■  Navigation  tool  to  find  the  right  process  at  the  right 
time 


-Views  provide  process  context, 
“What-to-do”  and  “How-to-do”  it 

-Lifecycle  process  mappings  tell 
the  user  when  to  execute  the 
“What-to-do’s”  and  “How-to-do’s” 
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Improving  Process  Management  with  a 
Process  Architecture 


■  Use  interface  information  to  identify  potential  impact  of 
process  updates 

-  Follow  I/O  threads  to  eliminate  unintended  consequences 

■  Use  contextual  views  to  identify  and  eliminate  areas  of 
redundancy 

■  Identify  high  leverage  areas  (e.g.  common  tasks) 

■  Improve  process  deployment  by  identifying  gaps  /  hidden 
areas 


o 


■  Improve  alignment  with  Business  Objectives 

-  Maintain  compliance  to  CMMI®,  AS9100 
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Keys  to  Success 


■  Understand  the  benefits  of  a  well-designed  Process 
Architecture 

■Align  your  Process  Architecture  with  your  business 
needs  and  objectives 

■  Develop  contextual  views  to 

-  Find  the  right  process  at  the  right  time 

-  Efficiently  improve  processes 

■  Maintain  your  Process  Architecture... Avoid  the  creep!!! 
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Why  The  Need  For  A  Standardized  Estimating  Tool? 

•  Need  defendable  Basis  of  Estimates  for  our  pursuits 

-  DCMA  requirement 

•  Requirement  for  bids  to  be  generated  with  a  consistent  and 
repeatable  process 

-  No  way  to  accurately  compare  bids  if  different  estimating  tools  and 
methods  are  used 

•  Estimation  models  must  be  calibrated  to  STS  past  program 
actuals  before  use  on  proposal  activities 

-  Use  of  standardized  tool  reduces  effort  required  for  calibration 

•  Level  of  knowledge  on  how  to  use  the  tools  varies  widely 
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What  Estimating  Tools  Did  STS  Evaluate? 

•  COCOMO  II  2000 

•  SEER-SEM 

•  TRUE-Planning  (True-S) 

•  REVIC  9.2 

•  SLIM 
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COCOMO  II  2000  Was  Selected 


•  Pros 

-  Latest  revision  of  the  COCOMO  II  model 

-  Extensive  documentation  to  the  model 

-  Model  factors  are  changeable  to  the  project 

-  Can  be  calibrated  to  STS  Program  Actuals 

-  Output  uses  standard  Excel  spreadsheets 

•  Cons 

-  Need  a  training  course  on  setting  up  and  running  the  program 

-  Requires  accurate  Source  Lines  Of  Code  (SLOC)  counts  for  each 
functional  area  to  be  estimated 

-  Many  complex  factors  to  learn  which  affect  output 

-  Embeds  non-engineering  functions  in  effort  estimates  (Program 
Office,  CM,  QA,  Manuals) 
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Calibration  Data  Basics 


What  did  you  do? 
Adapt/Reuse  SLOC 
New  Development  SLOC 
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How  long  did  it  take  (mo.), 
and  how  much  effort  (hrs.) 
ACTUALS 


Rockwell 

Collins 
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COCOMO  Calibration  Goals 

•  By  following  the  process... 

•  Calibrated  values  from  COCOMO  approach  Program  Actuals 
over  many  STS  programs 

•  Calibrated  COCOMO  model  becomes  the  standard  Software 
Estimating  tool  for  creating  bids  for  proposals  and  ECPs  for  STS 

•  Using  the  Calibrated  COCOMO  Template  establishes  a 
defendable  Basis  of  Estimate  (BOE)  for  written  narratives 
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We  Selected  COCOMO  II,  Now  What? 


•  Need  Data. ..And  lots  of  it!! 

•  Need  an  easy,  intuitive  way  to  capture  the  SLOC  from 
Programs 

•  Need  to  capture  Program  Actuals  to  build  a  calibration  file  with 
the  SLOC 
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What  Is  SLOC? 

•  SLOC  =  "Source  Lines  Of  Code" 

•  One  SLOC  is  one  logical  line  of  code  (an  If-Then-Else  is  one 
line  of  code) 

-  In  COCOMO  81,  known  as  Delivered  Source  Instructions 
(DSI).  One  If-Then-Else  =  X  lines  of  code 

•  Only  Source  lines  that  are  DELIVERED  as  part  of  the  product 
are  included  —  test  drivers  and  other  support  software  is 
excluded 

•  SOURCE  lines  are  created  by  the  project  staff  --  code  created 
bv  applications  generators  is  excluded 

•  Declarations  are  counted  as  SLOC 

•  Comments  are  not  counted  as  SLOC 
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Key  Terminology 

•  New  Code:  Completely  new  file  development 

-  Files  developed  from  scratch  for  the  program 

-  Not  ported  from  another  program 

•  Reused  Code:  Code  that  is  100%  reused  fNO  modifications) 

-  Files  taken  from  previous  program  baseline  w/o  change 

-  Files  taken  from  other  programs  w/o  change 

•  Adapted  Code:  Existing  code  that  will  be  modified  (adapted)  to 
work  for  the  objectives  of  the  program. 

-  Existing  files  that  will  be  modified  for  program  objectives 

-  Existing  files  that  will  delete  code  for  program  objectives 

-  Existing  files  that  will  add  code  from  other  sources 

-  Need  the  SLOC  size  before  any  modifications  are  made  (baseline) 
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Tools  To  Capture  New/Adapted  Code  Count 

•  :  Standard  STS  tool  for  computing  SLOC 

•  :  Alternate  tool  for  computing  SLOC 

•  Beyond  Compare:  Compare  files  for  differences  (New,  Deleted, 
Modified  lines  of  code) 

•  Perl  Script: 

-  In-house  tool  to  diff  files  or  directories. 

-  Outputs  Metrics  needed  for  collector 
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Collector  Forms 

•  Rockwell  Collins  STS  developed  the  SLOC  Collector  to  capture 
data  from  programs 

-  SLOC  Collector 

-  Project  Info  Collector 

-  Engineers  will  complete  form 

•  Both  forms  used  to  collect  project  SLOC  Actuals  and  COCOMO 
factors  for  completed  projects. 

-  Forms  are  embedded  into  the  Software  Development  Folders  (SDF) 
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153 

154 

Surface  Search 

Total  Subsystem  Size 

B520SS/IUS/TGP 

155 

156 

Header  File(s) 

TGP_Numeric_Entry_Page.h 

157 

TGP_Search_Page.h 

New  Code 
Example.  Only 
column  F  needs 

158 

TGP  Search  Table  Page.h 

159 

TGP_Searc1i_UfflSes_Page.1i 

160 

filled  out. 

TGP_SPI_Loading_Page.il 

161 

162 

Grayed  out  cells 

— 

AGWCP_0utput.h 

TGP_Shared.h 

163 

SHOULD  NOT  be 

filled  in. 
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27180  C 


c 

5? 

c 

81 

c 

59 

■  - 

C 

114 

c 

132  1 

10  4 

0.6 

c 

c 

4 

109  13 

118  5 

10  4 

10  4 

i 

0.6 

0.6 

1  1 
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A 


Subsystem 


File  Type 


File  Location/Name 


I0S 


Total  Subsystem  Size 
Header  File(s) 


/smt/avi  0/ 


Need  Total 
Subsystem  size 
BEFORE 
modifications! 

Adapted  Code 
Example.  Fill  in 
all  non-grayed 
out  cells. 

Grayed  out  cells 

SHOULD  NOT 

be  filled  in. 


reaponCommonUtils.l 
/WeaponLoadIC.h 


/MalfActive.cpp 
/MapDisplay.cpp 
/MgsLib.cpp _ 


/mgsTargetData.cpp 

/WeaponCommonUtils.cpp 

/v'VeaponLoadic.cpp 

/WeaponStatus.cpp 

/WpnConfig.cpp 

/WeaponLoadOff.cpp 

/Update_Wpn_IOS_data.c 
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D  !  E 

F  G  |  H  1  1 

,  j  1  k  ; 

L  M 

Primary 

Initial  Total 

Total  Language  New  Adapted  Modified  Reused 

IM  SU 

AA  UNFM 

SLOC 

SLOC* SLOC  ’ SLOC  "SLOC  W(%)" 

(%) 

-£t- 

25 

1 

30 

0 

0.2 

C++ 

4147 

600 

30 

Q 

0.2 

C++ 

■isgn 

d 

■5D 

n 

_n_2_ 

C++ 

7200 

760 

30 

Qi 

0.2 

C++ 

403 

50 

20 

2! 

0.2 

C++ 

jiffi.i-'c.'.iuL  _ia - 

114 

10 

20 

2 

0.2 

_ 

82 

1 

20 

0 

0.2 

^^+ 

95 

5 

20 

2 

0.2 

C++ 

166 

12 

30 

2 

0.2 

C++ 

850 

39 

30 

2 

0.2 

C++ 

684 

4, 

30 

0 

0.2 

C++ 

496 

1 

30 

0 

0.2 

C++ 

191 

1 

30 

0 

0.2 

C++ 

694 

19 

30 

2! 

0.2 

C++ 

3603 

71 

30 

8 

0.2 

C++ 

677 

4 

30 

6 

0.2 

C++ 

553 

49 

20 

i 

0.2 

C++ 

1707 

537 

_ i_  _  _J 

20 

8 

0.2 

C++ 

898 

180 

30 

8 

0.2 

C++ 

381 

7 

30 

4 

0.2 

C++ 

702 

6 

30 

4 

0.2 

C++ 

936 

1 

30 

2! 

0.2 

C++ 

1122 

2 

30 

2 

0.2 
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Project  Info  Collector 

•  Defines  the  Program  Effort  Adjustment  Factors  (EAFs) 

-  Can  be  defined  for  whole  project,  OR  tailored  per  subsystem 

•  Program  Attributes  are  program  wide 

•  Hover  help  available  for  all  entries 
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Project  EAFs 


A 


D  I 


I  H 


I 


Don't 

Effort  Multiplier  Attributes  Very  Low  Low  Nom  Hi  Very  Hi  XtraHi  Know 


Justification 


10 


19 

20 
21_ 

22 

23 


.  -  _|  _  -  _ 

Product  Required  SW  Reliability  (RELY) 
Database  Size  (DATA) 

_  Documentation  (DOCU) _ 

Product  Complexity  (CPLX) 
_ I  Develop  for  Reuse  (RUSE) 


_ I  _ 

Execution  Time  Constraint 
Platform  (TIME) 

Main  Storage  Constraint 
(STOR) 

_ |  Platform  Volatility  (PVOLL 


Personnel  Analyst  Capability  (ACAP) 

Programmer  Capability  (PCAP) 
Personnel  Continuity  (PCON) 
Applications  Experience 
(APEX) 

i 

Platform  Experience  (PLEX) 
LanguageJTool  Experience 
(LTEX) 


_ i _ §b| | 

Project  Use  of  SW  Tools  (TOOL) 


24 


Multi-site  Development  (SITE) 

X 

Required  Development 
Schedule  (SCED) 


The  effect  of 
software  failures  is 
lo w}  easily 
recoverable  losses. 
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Project  Attributes 


A 

B  |  C 

D 

E 

F 

\  f 

G 

H 

1 

1 

Don't 

2 

Project  Scale  Factors 

Very  Low  Low 

Nom 

Hi 

Very  Hi 

Xtra  Hi 

Know 

Justification 

3_ 

4 

Precedentness  (PREC) 

i 

5 

Devel  Flexibility  (FLEX) 

If  the  product  is  similar  to 
previously  developed 
projects,  PREC  is  high,  Use 
the  table  in  the  Instruction 

6 

Architecture/Risk 

Resolution  (RESL) 

7 

Team  Cohesion  (TEAM) 

guide  to  develop  the  PREC 
rating  for  your  program, 

... 

a 

Process  Maturity  (PMAT) 

9 

10 
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Creating  Calibration  File 

•  COCOMO  II  has  a  built  in  Calibration  method 

•  A  calibration  file  can  be  created  by  entering  actual  data  (SLOC 
and  Effort)  for  each  project 

•  The  project  can  be  characterized  by  any  number  of  modules 
representing  the  subsystems 

•  The  data  entered  should  reflect  the  actual  SLOC  produced 
(New,  Adapted,  Reused),  the  EAF's  associated  with  those 
subsystems,  and  the  Scale  Factors  for  the  Project  as  a  whole 
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■  USC-COCOMO  11.2000.0  -  Untitled 


|.Jn|@ 


File  Edit  View  Parameters  Calibrate  Phase  Maintenance  Help 

p|^|y|  *>Nml^l  jj 


Project  Name:  Project  1 


Scale  Factor 


Development  Model: 


Schedule 


Post  Architecture 


"3 


Module 

LJ 

1 

m 

at 

IP 

e 

tfOM 

Effort 

EST 

Effort 

INST 

K 

Modul  *  U 

me 

Size 

*h> 

EM 

L^iuriacm 

DEV 

DEV 

PROD 

COST 

COST 

5t-a££ 

DISK 

Mo  dul e  1 

S : 26250 

.  00 

0.32 

C 

90. 5 

29.0 

906.3 

o 

O 

i 

o 

o 

o 

2.0 

o 

o 

jfe. 

Mo  dul e  2 

S: 52500 

I 

.  00 

0.32 

c 

1S1. 0 

57.9 

906.3 

o 

o 

o 

0.0 

4.0 

0.0 

Enter  the  ESLOC 
data,  project  Scale 
Factors,  and  EAF  for 
all  modules 
comprising  the 
project. 


Lines 
le : 


78750 


Estimated 
Optimistic 
Most  Likely 
Pessimistic 


Effort 

Sched 

PROD 

COST 

INST 

Staff 

RISK 

69.5 

13.4 

1132. 8 

o 

o 

i 

o 

o 

o 

5.2 

86.9 

14.3 

906.3 

o 

o 

o 

o 

o 

6.1 

o 

o 

108.6 

15.4 

725.0 

o 

o 

o 

o 

o 

7.1 

3 


Ready 
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Enter  all  data  for  each 
subsystem  as 
Equivalent  SLOC 
(ESLOC),  as  calculated 
from  the  collector  form. 
(REVL=0  for  completed 
projects) 
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Project  Effort  Actuals 

•  Once  a  project  has  been  defined,  the  effort  and  schedule 
actuals  for  that  project  need  to  be  entered 

•  This  is  done  by  selecting  the  Project  selection  from  the 
Calibrate  pull-down 

•  The  inputs  are  Effort  and  Schedule 

-  Effort  is  defined  in  Person-Months  (PM) 

-  Schedule  is  input  in  Months  duration 
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Enter  the  actual 
Project  Effort  in 
Person-Months, 
and  Schedule  in 
Months. 


When  completed 
with  Project  1 , 
select  the  Cancel 
button. 
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Repeat  those 
same  steps  for 
each  project. 


End  result  will  be  a 
calibration  file 
composed  of  many 
STS  projects.  This 
will  produce  more 
accurate  effort 
estimates  based  on 
STS  actuals. 
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The  New  computed 
Coefficient(s)  for 
calibration  are  shown. 


Calibrated  Parameters 


After  selecting 

r plihrphnn  Mpfhnrl 

Compute,  select 

L-allUIGUUI  1  |t|CU  IUU 

*  Coefficients  only 

Coefficients  Only. 

Coefficients  and  Exponents 

1  1 

For  less  than  8  projects,  select 
the  cofficients  only  calibration 


Current 


Effort  Coefficient  2.94 
Schedule  Coefficient  3.67 


Effort  Exponent  0.91 
Schedule  Exponent  0.28 


New 

Effort  Coefficient  4.88 

Effort  Exponent 

0.91 

Schedule  Coefficient  3.97 

Schedule  Exponent 

0.28 

Accept 

Ignore 

Help  | 
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Calibration  Results 

•  Now  we  have  several  projects  for  calibration,  what  does  that 
tell  us? 
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A 

3 

c 

D 

E 

F  1  G 

H 

I 

J 

1 

1 

■ 

Actual 

Cocomo 

Cocomo  2000 

Cocomo  2000  Cocomo  2000  Calibrated  Calibrated 

Calibrated 

i 

Calibrated 

2 

Project 

PM 

2000 

Hans  +  Req 

Variance  Variance 

Cocomo 

Hans  +  Req 

Variance 

Variance 

3 

r 

PM 

PM 

PM  (%) 

PM 

PM 

PM 

% 

4_ 

5" 

1 

Project  1 

292.50 

-  ■_  _  ■■■ 

295.153 

315.814 

23.31  7.97% 

■  232.091 

301.837 

9.34 

3.19% 

6 

2  Project  2 

30.80 

30.895 

36.558 

5.76  7.13% 

77.315 

32.727 

1.93 

2.38% 

7 

3  Project  3 

5.97 

5.549 

5.937 

c _  ’ 

-0.03  -0.55% 

5.304 

5.675 

-0.29 

-4.94% 

1  1 

JL 

4  Project  4 

17.22 

7.273 

7.732 

-9.44  -54.81% 

6.951 

7.438 

-9.78 

-56.81% 

JL 

5  Project  5 

1939 

_  31.888 

34.120 

14.73  75.97% 

30.477 

_  32.610 

13.22 

68.18% 

10 

6 

Project  6 

52.67 

105.944 

113.360 

60.69  115.23% 

"  101.255 

1_  108.343 

55.67 

105.70% 

11 

7 

Project  7 

12 

8 

13 

Average 

1 

18.99  43.61% 

15.04 

40.20% 

14 

15 

! 

1 _ l _ 

Note:EM=152Hrs. 

16 

r 

\1 
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Coefficients 


Calibration  Coefficients 


Rockwell 

Collins 
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Rockwell 
Collins 

Variance  Explanations 

•  Data  Collection  on  1st  two  programs  done  imprecisely 

•  Initial  collector  forms  not  as  rigorous,  or  intuitive  as  current 
forms  (Updated  and  more  user  friendly) 

•  More  knowledge  garnered  from  USC  on  COCOMO  factors  and 
definitions  (flowed  into  Collectors) 

-  1st  several  projects  lacked  knowledge 

•  Project  4  mainly  a  HW  project  (SW  only  effort  extracted) 

•  Only  6  projects  (Really  need  >8  quality  data  projects) 

-  Several  more  projects  currently  in  work 


©  Copyright  2010  Rockwell  Collins,  Inc. 
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STS  COCOMO  Templates 


•  2  Templates  are  available: 

•  STS  COCOMO  Template  Calibrated 

-  STS  Calibrated  coefficients  are  part  of  this  file 

-  Effort  Factors  tailored  to  typical  STS  project 

-  Examples  of  types  of  code  estimates  shown 

•  STS  COCOMO  Template  Uncalibrated 

-  Only  Effort  Factors  tailored  to  typical  STS  project 

-  Uses  out-of-the-box  calibration  coefficients 

-  Examples  of  types  of  code  estimates  shown 


©  Copyright  2010  Rockwell  Collins,  Inc. 
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Rockwell 
Collins 

Summary 

•  STS  SW  Estimating  getting  better 

-  More  programs  needed  for  calibration 

-  End  Users  more  knowledgeable  in  completing  forms 

•  Collector  Forms  still  evolving 

-  Still  fairly  complex  even  with  additional  work 

-  Becoming  easier  to  populate 

•  Data  flowed  into  database  to  auto-generate  COCOMO  project 
file 

-  Generate  estimates  faster 


©  Copyright  2010  Rockwell  Collins,  Inc. 
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Rockwell 
Collins 

Lessons  Learned 

•  COCOMO  model  and  factors  very  complex 

-  Definitions  of  factors  hard  to  understand 

-  End  user  knowledge  varies  across  programs 

•  Engineers/PMs  perceive  non-value  added  work 

-  Collecting  SLOC  still  an  uphill  battle  with  Projects 

•  Automation  only  minimally  helps 

•  "What's  the  charge  #?" 

-  See  no  immediate  benefit  to  them 

•  WBS  structure  not  aligned  with  model  to  accurately  capture 
actuals 

-  Added  analysis  time  to  align  w/model 

-  May  lead  to  errors  in  effort 

•  Collectors  help,  but  are  not  perfect. 

-  Collectors  improved  with  feedback  from  end  users 

-  Data  collection  more  thorough  now 


©  Copyright  2010  Rockwell  Collins,  Inc. 
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Rockwell 
Collins 

Recommendations 

•  Standardized  Tool  Needed 

-  Pick  one  and  stick  to  it 

-  Become  educated  on  complexity  of  tool 

•  Single  focal  point  for  collecting  SLOC  and  Actuals 

-  Keep  out  of  engineers  hands 

-  Speeds  process  of  creating  calibration  projects 

•  Get  Buy-in  from  all  functional  areas  on  benefits 

-  Why  does  it  benefit  the  corporation?  ->  More  accurate  estimates. 

•  Capture  costs  in  way  that  aligns  with  SW  Estimating  model 

-  Align  WBS  with  model  to  accurately  capture  actuals 


©  Copyright  2010  Rockwell  Collins,  Inc. 
All  rights  reserved. 


34 


Rockwell 

Collins 


©  Copyright  2010  Rockwell  Collins,  Inc. 
All  rights  reserved. 


35 


NDIA  1 0th  Innnl  ClftUMI 
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“How  MBE  can  support 
Requirements  Development  and 
Technical  Solution.” 

November  18,  2010 


Frank  Salvatore 

High  Performance  Technologies,  inc. 

3159  Schrader  Road 

Dover  NJ,  07801 

(973)  442-6436  ext  249 
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HPTi  Proprietary  Information 


Outline 


>  Current  State  of  Practice 

>  Improving  the  State  of  Practice 

>  Vision,  Definitions 

>  How  MBSE  can  support  RD  and  TS 

>  Planning  Consideration 

>  Conclusion 


HPTi  Proprietary  Information 


Current  State  off  Practice 

l — ii — •  i  ; 

■  Requirements  are  captured  in  database  applications 

>  They  are  often  not  accurate  or  complete 

>  They  take  a  long  time  to  develop 

>  They  are  published  and  viewed  as  documents 

>  It  is  difficult  to  achieve  IPT  consensus 

■  Technical  Solution  is  captured  in  various  forms 

>  CAD/CAE,  PowerPoint  Slides,  Simulations,  and 
assembled  in  paper  based  documents. 

■  Often  traceability  is  weak  and  hard  to  maintain. 

■  Our  current  engineering  practices  lack  the  rigor  and  discipline 
necessary  to  be  explicit 
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Is  Tliere  a  better  Way? 


What  if  the  Requirements  and  Technical  Solutions  were 
all  captured  in  one  location  and  everything  was 
traceable? 

What  if  we  could  validate  our  requirements  at  the  same 
time  we  are  developing  our  technical  solution? 

What  if  the  IPT  could  reach  agreement  to  what  the 
requirements  are  and  what  they  mean. 

Can  a  Model  Based  Engineering  Approach  Help? 


“You  cannot  engineer  something 
if  you  cannot  see  it.” 

Scott  Workinger 


Vision  for  ii o  del  Based 
Engineering  (MBE) 


DRAFT 


A 

*  >f 

«n^iH 

He*"'1" 

♦<>* 

p 


«**•*,*  pfI£  S*^ 

. 


Program 

Management  Test 


Engineering  : 


Hardware 


Operational  Needs 
Current  Capabilities 
Budget/Schedule 


Software 


Configuration 

Management 


DRAFT 


Graphic  developed  by  the  NDIA 
Model  Based  Engineering 
Subcommittee. 
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MilSE  -  General  Definition 


It  is  about  System  Modeling 

>  System  Model  is  a  cohesive,  unambiguous 
representation  of  what  the  System  js  and  does. 

It  provides  a  description  of 

>  Requirements  and 

>  Technical  Solution  and 

Operational  Scenarios 
System  Behavior  (including  I/O) 

Physical  Architecture  (Structure,  interfaces) 

Dynamic  Simulation  (requires  “executable”  models) 

>  Verification  Procedures 

MBSE  is  used  to  produce  SE  products 

It  requires  a  Modeling  Language  (SysML)  that  is 
computer  interpretable 


Past 


Document  centric 


Minimum 
Required  to 
Define  System 
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Descriptive  Modeling 
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SysML  Overview 


■  General  Purpose  Graphical 
Modeling 

>  Structure 

>  Behavior 

>  Requirements 

>  Parametric 

■  Supports:  specification, 
analysis,  design, 
verification  and  validation 

■  Supports  model  and  data 
interchange  via  XMI  and 
the  evolving  AP233 
standard  (in-process) 


SysML  is  Derived  from  UML 


Removed  Diagrams 


Deployment  (Behavior) 

Object  (Behavior) 

Component  (Behavior) 

Interaction(Structure) 

Communication 

(Structure) 

Timing  (Structure) 


S yiffl.  DJ 


Name  Change 

•  Class  -  Block  Definition 

•  Composite  -  Internal 
Block 


I 


f 

[ 

t  * 

Requl  rartieni  \ 

St/urium 

Dug  ram 

Diagram  ^ 

PSfiQf&M 

Acfwrty 

Diagram 

Srquanei 

Diagram 

State  Machina 
Diagram 

Use  Case 
Diagram 

Same  as  UML  2 


1  I 

\  P»rim#tric 
■  Dies  ram 


Modified  fnwn  UML  2 


New  diagram  type 


Eleeh  Definidcn 

Internal  Bloch 

Diagram 

Diagram 

Diagram 
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How  can  MBSE 
sup|;j  urt  RO  an  d  TS 
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CMMI  VI  .1  Continuous 

Representation 
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SGI  -  Develop  Customer 
Requirements 


■  SP1.1-1  Collect  Stakeholder  Needs 
•  SP1.1-2  Elicit  Needs 

1  SP1.2-1  Develop  the  Customer  Requirements 


SG-2  Develop  Product 
Requirements 


SG-3  Analyze  and  Validate 
Requirements 


■  SP2.1-1  Establish  Product  and  Product  Component  Requirements 

■  SP2.2-1  Allocate  Product  Component  Requirements 
1  SP2.3-1  ID  Interface  Requirements 


i5 


1  SP2.1-1  Establish  Operation  Concepts  and  Scenarios 
1  SP3.2-1  Establish  a  Definition  of  Required  Functionality 

■  SP3.3-1  Analyze  Requirements 

■  SP3.4-3  Analyze  Requirements  to  Achieve  balance 
1  SP3.5-1  Validate  Requirements 

1  SP3.5-2  Validate  Requirements  with  Comprehensive  Methods. 


C 

o 

"h— * 

o 

C/3 

0 

O 


O 
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SG-1  Select  Product  -  Component 
Solutions 


SP  1.1-1  Develop  Alternative  Solutions  and  Selection  Criteria 
SP  1.1-2  Develop  Detailed  Alternative  Solutions  and  Selection  Criteria 
SP  1 .2-2  Evolve  Operational  Concepts  and  Scenarios 
SP  1.3-1  Select  Product  Component  Solutions 


SG-2  Develop  the  Design 


SP  2.1-1  Design  the  Product  or  Product  Component 
SP  2.2-3  Establish  a  Technical  Data  Package 
SP  2.3-1  Establish  Interface  Descriptions 
SP  2.3-3  Design  Interfaces  using  Criteria 


SG-3  Implement  the  Product 
Design 


SP  3.1-1  Implement  the  Design 

SP  3.2-1  Develop  Product  Support  Documentation. 
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Requirements  Development: 

$G-1  UyyyJ'jp  Customer  Requirements 

l — ii — •  i  ; 


■  Collect  Stakeholder  needs 
(SP  1.1-1)  using  the 
Requirements  model 
element 

■  Or  capture  them  in  a 
requirements  database  and 
synch  them  with  the  model 

■  SysML  can  be  used  to 
capture  and  manage 
relationships  between 
requirements 

■  Auto  Generate 
Requirements  Trace 
Matrices  and  Specifications 


ID 

Name 

Specification 

SR  0001 

^  StakeholderReql 

Req  Text  for  SRI 

SR  0002 

2J  StakeholderReq2 

Req  TeKt  for  SR2 

SR  0003 

2j  StakeholderReq3 

Req  Text  for  SR3 

req  [Package]  MyRequirementsDiagrams  [Requirements  Trace  Diagram]  J 


MyRequirement 


MyFunctionalRecI 


43 


MyProductReqs 


MyStakeholderRecj 


^ ^ — W 


>■ 


_ m\ 

■jfleqilremeit 
F  u  n  cti  o  n  a  I R  e  q  1 


Captured  Stakeholder 
Requirements 


HPTi  Proprietary  Information 


ll 


Requirements  Development: 

SG-1  Dew y Jap  Customer  Requirements 


■  Use  “Use  Case  Analysis”  to 
capture  capabilities  and 
elicit  requirements  from 
stakeholders  (SP1.1-1,  SP 
1.1-2) 

■  Trace  Requirements  to  Use 
Cases 

■  Seeing  requirements  in  a 
diagram  tends  to  draw  out 
requirements 


Diagramming  and  visually  presenting  what  has 
been  captured  (SP  1.1-1) 
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Requirements  Development: 

SG-1  Dew y Jap  Customer  Requirements 


■  Use  Block  Definition 
Diagrams  (BDD)  to  capture 
where  your  system  of  interest 
fits,  defining  the  Physical 
System  Boundary  (TS-SP  1.1- 
2, 1.2-2) 

■  Perform  Domain  Analysis  to 
understand  context  and 
further  drive  out  a  clearer 
understanding  of  the  problem 
that  needs  to  be  solved. 

■  Drive  out  external  interfaces 


Diagramming  and  visually  presenting  what  has 
been  captured  (SP  1.1-1) 
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Update  use  cases  and  Requirements  as  you  learn  more  n 


Requirements  Development: 

SG-3  Analyze  and  Validate  Requirements 


■  Model  Domain  Activities  with 
Activity  Diagram  (ACT)  to  Elicit 
needs  and  to  capture  behavior 
(SP  1.1-2) 

"  It  starts  to  capture  what  the 
operational  concepts  and 
scenarios  are  (SP  3.1-1) 

■  Provides  a  precise  definition  of 
required  functionality  (SP3.2-1) 

■  It  serves  the  purpose  of 
Analyzing  and  validating  the 
requirements  (SP3.3-1,  SP3.5-1, 
SP3.5-2) 

■  Establishes  Functional  System 
Boundary 
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Update  everything  as  you  learn  more. 
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Requirements  Development: 

SG-3  Analyze  and  Validate  (Requirements 


■  Use  an  Internal  Block 
Diagram  (IBD)  to  elicit 
requirements  and  identify 
interface  requirements  (SP 
1.1-2,  2.3-1). 

■  Rigorous  and  explicit 
capture  and  documentation 
of  external  interfaces. 

■  Serves  the  purpose  of 
Analyzing  and  validating  the 
interface  requirements  (SG 
3). 

■  Merge  of  physical  and 
functional  system  boundary 
definition 
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Update  everything  as  you  learn  more. 


15 


Requirements  D  evelopment: 

SG-2  UavyJop  Product  Acquirements 


Identify  behaviors  and 
derive  functional 
requirements  of  your 
system  and  system 
components  (SP  3.2-1) 

Use  IBD  to  capture  the 
arrangement  and 
association  between  a 
selected  system  solution 
alternative. 

Identify  Interface 
Requirements  (SP  2.3-1) 


Input  from  Technical  Solution  (SP1 .1  -2,  SP2.1  -1 ) 

act  [Use  Case]  UseCase2  [activity  2]  } 


System  Functions 


[that  way] 


Do  this  other 
thing  based 
on  decision 


Do  this  before  you  know  what  the 
components  are 


ST 


Identify 

Component 

Interactions 


Update  everything  as  you  learn  more. 


1  itsPiece2:Piece2 


1  itsPieceliPiecel 


1  itsPiece3:Piece3 

1  itsPiece4:Piece4 

System  Internal  Interfaces 


This  would  be  done  once  you  know  what 
your  components  are 
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Requirements  Developments 

S6*2  DyyyJyp  Product  ftequireaneuts 


■  Capture  the  Systems 
Physical  Structure  or 
Hierarchy  with  a  Block 
Definition  Diagram  (BDD). 

■  If  you  know  what  these  are 
begin  Developing  Product 
Requirements  (SP2.1-1). 

■  Else  perform  Technical 
Solution  Practices. 

■  Identify  associations  and 
quantities. 

■  Allocate  requirements  to 
each  system 

component/block  (SP2.2-1) 


Input  from  Technical  Solution  (SP1.1-1,SP  2.1-1) 

bdd  [Package]  MyStructureDiagrams  [System  Hierarchy]  ) 


Identify  System  components 
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Update  everything  as  you  learn  more. 


Requirements  Developments 

SG-2  DyyyJyp  Product  ftequireaneuts 


■  Use  Sequence  Diagrams, 
Activity  Diagrams  and  State 
Machine  Diagrams  to 
Identify  behaviors  and 
derive  functional 
requirements  of  your 
system  and  system 
components  (SP  3.2-1) 

■  This  should  serve  as  the 
authoritative  source  for 
describing  design  details 

■  Requirements  can  be 
directly  traced  to  design 
elements  in  the  model 

Results  in  a  Functional  Architecture 


Input  from  Technical  Solution  (SP1 .1  -2,  SP2.1  -1 ) 
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Requirements  Developments 

SG-2  DyyyJyp  Product  Requirements 


■  Allocate  Product  Component 
Requirements  to  System 
Elements  (SP2.2-1). 

■  This  requires  execution  of  the 
Technical  Solution  Process 
Area 

■  If  modeled  properly  a  change 
to  the  model  elements  will 
invoke  a  change  to  all 
diagrams  and  uses  of  that 
model  element 


Input  from  Technical  Solution  (SP1 .1  -2,  SP2.1  -1 ) 


To::  block  Scope:  MyB  locks 


0  MySystem 

0  ExtemalSysteml 

ExtemalSystem2 

8  ES2 Block1 

m  ES2 Block2 

0  MySystem  Domain 

0  Piece  1 

0  Piece2 

0  PieceS 

8  Piet 

(g)  activity  J) 

(g)  activity 2 

^  Piece2 

Results  in  a  Physical  Architecture  19 
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Requirements  D  evelopment: 

SG-2  UavyJop  Product  Acquirements 


Use  a  BDD  and  Parametric 
Diagrams  to  show  allocation 
of  product  component 
requirements  and  to  capture 
Design  Constraints  (SP  1.1- 
1,  SP-2.2-1). 

Used  in  to  define  design 
trade  offs  (SP  2.4-3)  and  to 
balance  the  requirements 
(SP3.4-3) 

Parametric  Diagrams 
capture  how  requirements 
can  be  analyzed  (SP  3.3-1) 
for  validation  (SP3.5-1)  and 
evaluate  performance 
measures  (PA  EV). 


HPTi  Proprietary  Information 


Input  from 
Technical 
Solution 
(SP1.1-2) 


par  [Package]  MyParametricsDiagrams  [Sample  Parametrics  Di 


bdd  [Package]  MyStructureDiagrams  [Analysis  of  Parametrics]  ~J~ 


.block, SystemOflnterest* 

MyBlocks::MySystem 


0  Weight:  kg 


.block* 

Basyncjness 

MyBlocks::Piece1 

Attributes 

Operations 

.block* 

MyBloc  ks : :  Piece2 

.block* 

MyBlocks::Piece3 

.block* 

MyBlocks::Piece4 

Attributes 

Attributes 

Attributes 

0  Weight:kq 

0  Weight:kg 

S  Weiqht:kq 

Operations 

Operations 

Operations 

WeightEquation 


Constraints 

{e)total_weight 


ConstraintParameters 
i  @Total:kg  i 
[*]  weighting 
@weight2:kg 
@weight3:kg 
@weight4:kg 


«  Attribute* 

MyBlocks::Piece1.Weight:kg 


weight1:k< 


WeightLalc:Weighttquation 


Attributes 

< 

-  □ _ 

c 

Constraints 

□ 

total_weight 

> □ □ 

MyBlocks::MySystem.Weight:kg 


MyBlocks::Piece2.Weight:kg 


My  Blocks::My  System. itsPiece1:Piece1 


S  Weight:  k 


MyBlocks::Piece4.Weight:kg 


.Attribute* 

MyBlocks::Piece3.Weight:kg 


MyBlocks::My  System.  itsPiece2:Piece2 


S  Weight:  kg 


S  sync_message():void 
y  reply  to  sync_message():void 
8async_message() 
y  reflexive  message():void 


My  Blocks::My  System.  itsPiece3:Piece3 


S  Weight:  kg 


MyBlocks::My  System.  itsPiece4:Piece4 


B  Weight:kg 


Work  with  the  IPT  and  get  them  to  agree 
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Requirements  DjveJopment: 

SG-3  Analyze  ami  Validate  Requirements 


■  Use  Requirements  Model 
Elements  to  capture  Test 
Case  Descriptions 

■  Use  Requirements 
Diagrams  to  trace  between 
Test  Cases  and 
Requirements. 

■  A  diagram  of  this  may  help 
the  team  relate  to  how 
many  tests  are  necessary  to 
verify  a  requirements  and 
vice  versa. 


■  Auto  generation  of 
Verification  Matrix 


EJ  Functional  Reql 

EJ  Functional  Req2 

E|  StakeholderReql 

/„  Test  Case  One 

i 

Functional  Req2 

All  Requirements  should  have  Associated  Verification  Methods 
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Technical  Solution: 

SG-1  Select  Product  -  Component  Solutions 


l — ii — '  i  ; 

■  You  will  need  to  construct 
behavior,  cost,  architecture, 
physical  mock  ups, 
prototypes  of  alternative 
solutions. 

■  Using  SysML  to  capture  a 
description  to  each  alternative 
can  help  to  communicate  with 
the  IPT 

■  The  Decisions  can  be 
captured  as  notes  in  SysML 

■  Evaluation  Criteria  will  likely 
emerge  as  requirements  so 
capture  them  as  such. 


iftg  If'jaj.jg]  ti'tif:  j-j/jmr .|Pp.iwim*nl;  Trj^:»  t’upumj 


rz 


r-i|i«*-i>L  1 


Use  to  establish 
Decision  Criteria 


.  1  :  '  • , 


4- 


M*£*4rtl  I^SVdifn 


1 

hr1 _ bi  l. 


Fjc*3  i,Vnpn  Kp 


'  Hyllliitki::Uy^rnMin.iLilJnrt*J:Ktotii3 

-  IfyRkuti:  U>  Sj.iwfli.laJ'im.-fll  V\tnfA 

B  Wa*0»i  dg 

§  ****** 

9  **»**>« 

I®  iyra:_fi**W9»ri’M¥i 

irihiM  lArtiJgatiiiM 
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Decision  Criteria  should  be  consistent  with  the  model 


Technical  Solution: 

SG-2  UavaJ yp  iha  Design 


■  Use  a  BDD  to  allocate 
requirements  from  the  logical 
design  to  each  design  Alternative 
(SP  1.1-2) 

■  Use  BDD  and  IBD  to  capture  an 
abstraction  of  the  Physical  Design. 
This  is  the  Product  Architecture 
and  the  authoritative  description 
of  the  design. 

■  The  whole  process  is  recursive  till 
you  are  at  the  lowest  level  in  the 
system  hierarchy 

■  The  System  model  will  organize 
your  product  description  data 
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Include  the  model  with  your  technical  data  package 


23 


Technical  Solution: 

SG-3  Implement  the  r'/'j'luct  Design 


For  SW  products  SysML  and  UML  can 
be  used  to  auto  generate  Code. 

For  Electrical  and  Mechanical  Systems 
CAD  product  models  can  be  used  to 
directly  support  manufacturing, 
assembly,  inspection,  and  test. 

A  rigorously  modeled  design  will 
support  resolving  problems  as  they 
come  up. 


Model  Based  Engineering  Can  Help 


Planning  Considerations 


■  Use  people  who  are  open  to  this  approach  and  are  willing  to  go  the 
extra  mile.  Pair  them  with  someone  who  is  experienced.  It  will 
accelerate  the  learning  process  and  keep  them  from  stalling  out. 

"  Conduct  a  Gap  Analysis 

■  You  will  need  to  pick  a  tool 

■  You  will  need  a  Methodology  that  describes  how,  why,  when,  and 
what  to  model. 

■  You  will  need  to  train  even  the  most  experienced  Systems  Engineers 
(Language,  Tool ,  Methodology) 

■  You  will  want  to  pilot  it  on  several  projects  to  learn  your  own  lessons. 

■  Keep  it  simple  at  first  and  expand  to  more  complex  modeling 
problems. 

■  Model  what  you  know  best  and  see  if  everyone  really  understands  it 
the  same  way.  (You  will  need  to  work  towards  this.) 


HPTi  Proprietary  Information 
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Conclusion 


■  Established  the  need  to  change 

■  Discussed  how  Modeling  supports 
many  of  the  RD  and  TS  CMMI 
practices. 

■  It  will  take  time  and  requires 
training  and  practice. 

■  We  can’t  keep  operating  the  same 
way. 


HPTi  Proprietary  Information 
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Agenda:  Using  the  Agile  Methodology  to  Mitigate 
sei Partner  pjs|^s  Qf  Highly  Adaptive  Projects 


Risks  in  Highly  Adaptive  Projects 
Typical  Management  Scenarios 
Agile  Software  Development  Methodology 
Steps  for  Implementing  CMMI-DEV  Practices  in 
Agile  Projects 

Strategy  Map  for  Using  Agile  Practices 
Benefits 

Lessons  Learned 
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—  Agile  Methodology  for  Risk  Management 
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Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Using  the  Agile  Methodology  to  Manage  Risks  of  Highly  Adaptive  Projects 

Dynamic  Requirements 

Impact  of  customer  co-location  on 
requirements  management  & 
tasking 

Aggressive  Schedule 

Immediate  effect  of  changing 
customer  priorities  &  direction 

Limited  Resources 

Typical  work  performed  on  a  fixed 
LOE  or  time  &  material  basis 

Risks  In  Technology  &  Support  Projects 


SEI  Partner 


The  requirements  are  ALWAYS  changing. 


Time  and  schedules  are  ALWAYS  aggressive. 


Resources  will  ALWAYS  be  tight. 


To  mitigate  these  risks  you  need  processes  that 
address  these  risks  to  allow  rapid  adaptation. 


»=  Scenario  for  Risk  Management 

SEl Partner  Practices  in  Small  Businesses 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Small  businesses  often  function  in  focused  markets  as: 

•  Suppliers  of  specialized  on-site 

technical  services,  key  personnel,  applications 
or  products 

•  Domain  specialists 

•  Staff  augmentation 

These  focused  markets  may  not  provide  needed 
revenue  or  market  share  growth  to  readily  prosper  in 
today’s  economy. 


Costs  of  improving  project  and  risk  management 
practices  are  considered  key  investments  in  these 
vulnerable  cash  flow  environments. 


Both  near  and  long  term  cash  flow  analysis  typically 
includes  factors  such  as: 

•  Customer  base  stability  &  revenue  cycles 

•  Projected  revenue  growth 

•  Costs  of  acquiring  appropriate  resources  to  improve 
management  practices,  such  as  training,  process 
engineers  and  collaborative  software  and  hardware. 


SEI  Partner 


Scenario  for  Implementing  Risk 
Management  Practices  in  Small 
Businesses  (continued) 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Understanding  and  improving  project  management  and  risk 

management  practices  is  a  key  factor  in  revenue  growth. 

Planning  improvements  typically  includes  assessment  of 

critical  factors  such  as: 

•  Required  resources  (available  staff,  feasible  schedule, 
platforms,  facilities,  collaborative  tools) 

•  Past  performance  data  (costs,  schedules,  lessons 
learned) 

•  Training  in  the  domain  and  software  development 
method  (Agile) 

•  Risk  identification  and  impact  assessment 

•  Risk  mitigation 


A  practical  knowledge  base  useful  for  improving 
management  practices  can  be  provided  by  process  models, 
such  as  CMMI  and  ISO  9000.  Obtaining  CMMI  benchmarks 
and  ISO  9000  certifications  are  often  stepping  stones  to 
expand  government  and  commercial  customers. 

Selecting  and  implementing  any  process  and  life  cycle 
models  is  both  costly  and  can  be  risky  as  there  are  no 
guarantees  of  success  on-site  with  the  customer. 
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Scenario  for  Implementing  Risk 
~  Management  in  Small  -  Medium 
SEIPartner  Businesses  (continued) 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Planning  a  process  improvement  strategy  typically  includes 
identifying  problematic  projects  with  typical  issues  such  as: 

•  Limited  resources  and  ambitious  schedules 

•  Lack  of  experience  and  staff  training 

•  Excessive  staff  turnover  rate 

•  Stability  of  current  customer  base 

•  Projection  of  market  growth 

•  Over  commitment  of  key  staff 

•  Inaccurate  planning  data  for  costs  and  schedule 


Implementing  a  process  improvement  strategy,  small  businesses 
should  customize  a  path  of  small,  adjustable  steps. 

An  example  of  a  path  of  small  steps  is  a  hybrid  approach  of 
improving  project  and  risk  management  practices  in  CMMI-DEV  first. 


The  next  step  is  a  selection  of  a  life  cycle  model  that  is  suitable  for 
the  work  requirements,  location  and  level  of  customer  participation. 


Building  Bridges  Using  CMMI-DEV  and  Agile  to 
SEl  Partner  Manage  Risk 


— 

Daily  Scrum 
Meeting 


CMMI  Is  Designed  to  Be  Tailored 
to  Your  Business  Environment. 

S  Fully  align  process  improvement  with 
your  business  goals 
S  Look  at  what  you  are  actually  doing, 
s  If  a  “good  fit”  -  document  and  follow 
S  Leverage  off  existing  activities  -  e.g. 
Agile 

S  Promote  maximum  flexibility  and 
efficiency  in  your  process 
improvement  approach 
S  Emphasize  measurement  and  analysis 


Agile  life  cycle  model 


Copyright  ©  2005,  Mountain  Boat  Software 


Product 

Backlog 


/ - \ 

Potentially 

Shippable 

PRODUCT 

Increment 


— 

Sprint 

Backlog 


SEI  Partner 


Key  Process  Improvement  Factors  &  Sources 
of  Risks 


process  improvement  factors  introduce  sources  of  risks  to  developers! 


Training 


Resources 


Business  Case 


Contract  Type 


Project 

Planning 


Communication 


Domain 

Experience 


Life  Cycle  Models 


Processes 


Choosing  an  incompatible  life  cycle  development  model 
impacts  all  other  key  process  improvement  factors. 


All  KEY 


C  cm 


SEI  Partner 


Sources  of  Risks  in  Life  Cycle  Models 


Waterfall 


cycle  times 
impact  of  changing  requirements 


direct  end  user  involvement 


Spiral 


cycle  times 


CJ 


impact  of  changing  requirements 


direct  end  user  involvement 


Agile 

inadequate  training 
customer  awareness 
management  advocac 


- . 
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Practice  Implementation  Across  Small  Organizations 


Organization 


Framework  of  the  CMMI-DEV  model 
provides: 

1.  Increases  in  implementation 
efficiency  due  to  redundancy 
in  model  functions,  such  as 
Project  Management  &  Support 
Process  Areas  and  generic  practices. 

2.  Minimal  opportunities  for 
conflicts  with  customer,  staff  and 
resources  with  adequate  model 
understanding 


CMMI 

Process 

Management 


CMMI 

Engineering 


CMMI 

Project 

Management 


i  CMMI 
Support 
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Steps  for  Implementing  Risk  Management 
with  CMMI-DEV&  Agile 


Lessons  Learned 

^What  did  we  do  well? 
^Where  were  we  lucky? 
^What  do  we  need  to 
improve? 


Develop 

Agile  project  plan 


Understand  the  Risks 


market  area 
revenue/profit 
projected  growth 


Key  growth  Investments: 

Process  &  Agile  training 
Collaborative  H/W  &  S/W 
Process  engineer  support 


\ 


Impact 

Metrics 


Cost 
Schedule 
Technical 


Identify  risks: 

•project  specific 
•technical  impacts 
•mitigation  plans 
customer  involvement  Template 


High 

Medium 

Low 

Risk 


Evaluate  business  opportunity 
“Risk  of  winning” 


winning 

Contract  type 
New  customer 
Technical  complexity 
Staff 


-*■  Strategy  Map  for  Implementing  Risk  Management 

SEI Partner  Usjng  Agj|e  gnd  CMM|.DEV  Xk 


Dynamic 

requirements 


Aggressive 

schedule 


Agile  Method 

Engineering 

Techniques 


CMMI-DEV 
Engineering 
Management 
Support 


Risk 

management 

tools 


Scrums 


Burn  Down  (MS  Excel) 


Quality 

Assurance 


White  Boards 
Peer  Reviews  - 
Impromptu  &  Team 
Programming 


Configuration 

Management 


Structured 

Decisions 
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Risk  Reduction  Solution  -  Process  Improvement 


Risk  reduction  is 
NOT  fighting  fires 
or  “just  in  time 
solutions.” 


Reduce  variation. 


*  It  is  simply  a  way  of  looking  at  how  we  can  do 
our  work  better. 

•  A  series  of  actions  taken  to  identify,  analyze 
and  improve  existing  processes. 

•  These  actions  often  follow  a  specific 
methodology  or  strategy  to  create  successful 
results. 


*  Remove  activities  that  have  no  value  to 
the  organization. 

*  Improve  customer  satisfaction. 

*  Process  improvement  is  important  as  it 
has  often  been  said  that  process 
accounts  for  80%  of  all  problems  while 
people  account  for  the  remaining  20%. 


Recognize  and 
document  best 
practices. 


•  Provide  lessons  learned. 

•  Encourage  staff  members 
to  contribute! 


-==-  What  are  Best  Practices  and  Benefits? 

SEI  Partner 


Best  practices  are  effective,  high-leverage  technical 
and  management  PROCESSES  that  have  been 
implemented  and  proven  on  successful  projects. 


Examples  of  Best  Practices 


Proven  Models  -  i.e.  CMMI-DEV, 
ISO  9001 

Project  Planning 

Measurement  and  Control 

Peer  Reviews 

Risk  Management 

Quality  Assurance 

Status  Reporting 

Configuration  Management 

Agile  Software  Development 


Benefits 


Risks  are  reduced  &  mitigated 

Repeatability  is  achieved 

Clarity  and  understanding  are 
increased 

Common  terminologies  and 
consistent  styles  are  developed 
and  followed 


sei Partner  Typical  “Roadblocks  and  Hurdles”  to  Process  Improvement 


Resistance  to  Change  Lack  of  Commitment  and 

Resources 


“We  tried  process 
improvement  before,  and  it 
didn’t  buy  us  anything.” 

“CMMI  /TQM/Agile/Lean  Six 
Sigma/  and  best  practices 
are  just  another  fad.” 


“Who’s  paying  for  it?  If  it  isn’t 
paid  for,  we’re  not  doing  it.” 

“I’m  retiring/transferring  from 
this  job  soon,  I  don’t  have  to 
get  involved.” 

Developers  don’t  have  the 
time  because 
much  rework. 


there  is  so 


AGILE  Software  Development  Methodology 


SEI  Partner 


s  Activities  are  known  as  'timeboxes1. 

S  Each  iteration  passes  through  a  full  software  development  cycle, 
including: 

planning, 

requirements  analysis. 

design, 

unit  tests.  Quality  Assurance  testing, 

coding  until  the  unit  tests  pass  and  a  working  product  is  finally 

demonstrated  to  stakeholders. 

S  “Documentation  is  no  different  than  software  design  and  coding  for 
assuring  repeatability  and  re-generation. 

S  Strong  team  discipline  is  required  to  code  for  agility. 

s  At  the  end  of  each  iteration,  stakeholders  re-evaluate  project  priorities 
with  a  view  to  optimizing  their  return  on  investment. 

S  Agile  emphasizes  face  to  face  communication  with  all 
stakeholders! 


PRODUCT 

5  PRINT 

BACKLOG 

Backlog 
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Agile  -  Navigating  Class  IV  Rapids  (AKA  Risk  Reduction) 


CMMI-DEV 


Agile 

Quality  Product 


□  Agile  development  methods  minimize  risk  by 
developing  software  in  multiple  'iterations'  of 
short  time  frames 

□  Each  iteration  passes  through  a  full  software 
development  cycle 

□  In  our  Agile  projects,  SCRUM  Boards, 

Burndown  charts  and  other  project  and  working 
artifacts  were  defined  and  identified  as  “CMMI 
artifacts”. 

□  Face  to  face  communication  &  EA  reviews 
ensure  peer  review. 

□  At  the  end  of  each  iteration,  stakeholders  re¬ 
evaluate  project  priorities  and  risks. 
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Agilizing  CMMI  -  SCRUM  and  CMMI 


S  Incorporating  CMMI  Processes 

Tailoring  processes  to  incorporate  Agile 
iterations 

Mapping  CMMI  SPs  to  Agile  iterations  and 
artifacts 

Monitoring  and  controlling  projects 
requirements  and  changes 
testing 
defects 

S  Using  Agile  at  our  shop 


Visual  Source  Safe 

Configuration  Management 
Source  Control 


Visual  Studio 

Development  Environment 
SQL  Management  Studio 

Development  environment 
Enterprise  Architect 

Requirements  management 
Requirements  traceability 

Axosoft  OnTime 

Defect  Management 


Agile  &  CMMI-DEV 
Processes 

(Process  Asset  Library) 


Defined  processes 
Scrums 
Scrum  Boards 
Sprints 

Burn  Down  -  (MS  Excel) 
White  Boards 
Peer  Reviews 
Team  programming 
Performance  metrics 


SEI  Partner 


Where  to  Start? 


•  Look  at  what  you  have  in  place  /  '  (\  *  ( 

-  What  are  you  doing  well  -  right  now  ' _ _ ^ 

•  Establish  your  SEPG 

•  Tailor  the  CMMI  process  descriptions  to  map  to  current  processes 

-  “By-in”  by  Software  Engineering  and  Management 

-  PAL 

•  Develop  processes  for  things  you  are  not  doing  now 

•  Discuss  with  Software  Engineering  Team 

•  Train  the  Team  ! 


•  Follow  up  on  how  processes  are  being  implemented  and 

•  Modify  if  necessary  to  get  a  “best  fit”  ! 


Process  Asset  Library  -  Processes  and  Standards 


SEI  Partner 


ary  (PAL)  -  Microsoft  Internet  Explorer  provided  by  DOECOE 


□rites  Tools  Help 

0  0  y  ■  ' 5earch 


* 


Favorites 


40 


plications/Process%20Asset%20Library%20PAL/Forms/AllItenns.aspx 


v  S  Go  Links  >y  '-^Convert  ^  (^Select 


^plications  Team  Web  Site 

i  Team  Web  Site 

:iIRN  Expansion  eUP  ePegasus 


Welcome  Robi 


This  List  v 


FY  Initiatives 


FCDS 


HR 


Maintenance  Middle  Tier  NCSO  NSO  Mission  Migration  P3  SMRE  PAMS2  TQP  WSU  CMMI  ML  3 


Applications  Team  Web  Site  >  Process  Asset  Library  (PAL) 

Process  Asset  Library  (PAL) 

Library  oF  project  plans,  process  descriptions,  Forms,  and  templates. 


New 

T  Upload  T  Actions  T  Settings  T 

View:  All  [ 

Type 

Name 

v  ModiFied 

.  ModiFied 

OJ 

Agile  ReFerences 

2/9/2009  12:02  PM 

Young,  Mic 

Ca 

Developer  Standards 

2/2/2010  2:43  PM 

Young,  Mic 

Ca 

External  Documents 

8/11/2008  9:26  AM 

Young,  Mic 

L_a 

Forms  and  Templates 

8/11/2008  9:14  AM 

Young,  Mic 

Cm 

In  Work 

8/26/2008  8:25  AM 

Young,  Mic 

Cm 

LiFecycle  Diagrams 

7/8/2009  1:52  PM 

Young,  Mic 

Cm 

Plans 

8/11/2008  9:13  AM 

Young,  Mic 

Process  Descriptions 

8/11/2008  9:14  AM 

Young,  Mic 

Cm 

Training  Materials 

12/29/2008  11:52  AM 

Young,  Mic 

NNSA  SC  Organizational  Process  DeFiniton  and  Tailoring  Guidelines 

9/10/2009  4:38  PM 

Roberson, 

d 

NNSA  SC  SDLC  Procedure2 

11/4/2008  11:11  AM 

Young,  Mic 

m 

PAL  Description 

12/17/2008  3:14  PM 

Roberson, 

Work  Environment  Standards  Description 

11/18/2008  4:16  PM 

Young,  Mic 
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Process  Descriptions  Based  on  how  We  Do  Business 


i 

PD_2.4_5print  Review  Meeting 

6/29/2009  11:53  AM 

Young,  Michael 

a 

PD_3 . 0_Requirements  Development 

//\ryj 

6/29/2009  10:42  AM 

Young,  Michael 

a 

PD_3,3_Manage  Requirements 

/X  J  \  ^ 
\\  \  \ 

6/29/2009  10:53  AM 

Young,  Michael 

a 

PD_4,0_Peer  Review 

7/1/2009  4:26  PM 

Young,  Michael 

a 

PD_5.0_Project  Support 

8/10/2009  3:28  PM 

Young,  Michael 

a 

PD_5.1_CM  Capability 

8/10/2009  1:31  PM 

Young,  Michael 

a 

PD_5 . 3_T rack  Change  Requests 

8/10/2009  1:31  PM 

Young,  Michael 

a 

PD_5,4_Manage  Baselines 

8/10/2009  1:32  PM 

Young,  Michael 

a 

PD_5 . 5_Conf  iguration  Audits 

8/10/2009  1:32  PM 

Young,  Michael 

a 

PD_Deployment 

5/11/2009  2:23  PM 

Young,  Michael 

a 

PD_PM_Lifecyde 

5/26/2009  3:13  PM 

Young,  Michael 

a 

PD_Release  Management 

10/26/2009  8:54  AM 

Young,  Michael 
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Process  Area  Description 


Enterprise  Information  Technology  Services 
Applications  Development  Process 


General  Information 


Process  Area:  RQDM  (Requirements  Specific  Goal:  (All  SGs) 

Development  and  Management)  (REQM/RD) 


Summary:This  process  outlines  how  the  department  elicits,  develops,  documents  manages,  and 
validates  requirements  for  a  software  development  project  The  department  utilises  Customer 
Requirement  Meetings,  and  the  Enterprise  Architect  tool  in  an  manner  integrated  with  the 

project  sprint  cycle  to  accomplish  these  objectives. 


Approval : 


Date : 


Process  Area  POC:  Joseph 
Robinson 


Revision  History: 


Revision!  Level 

O  ate 

Description 

Change  Summary 

PD_RDQM_2.  1  ,2_Requirements 

D  eve  1  o  p  m  e  nt 

5/26/09 

1  n  iti  a  1  d  ocu  rn  e  nt 

M/A, 

PD _ 3.0  Requirements  Development 

6/29/09 

Revised  names  and 
numbers  to  fit  new 
model. 

Process  Area  Overview 


Scope  :  ^Ch an g e  as  ne e d e d .  > 

This  process  area  applies  to  all  software  development  projects  under  the  responsibility  of  the  Information 
Technology  Department  (ITD)  Applications  Development  group 

Standards  -  Developed  by  TWG  within  the  SEPG 
available  to  all  developers  on  SharePoint  Site 
S  Work  Environment  Standards 
S  Web  Design  and  Usability  Guidelines 
s  Coding  Standards 


Process  Descriptions  -  Diagrams 
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Development 

Initial  Planning  Cycle 
Project  Backlog 
EA 

Management - 
Tracked  in  EA  through  testing 


Process  Descriptions  -  Mapping 
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Development 

Initial  Planning  Cycle 
Project  Backlog 
EA 


Management - 

Tracked  in  EA  through  testing  ) 


Sprint  Plan 

•  Testing  is  built 
into  each 
sprint 


Scrum 

•  Assign  tasks 
to  resources 

•  Reassign 
based  on 
burndown 


print  Planning 

Requirements 
Available 
resources 
Development  & 
QA 


Defect  Workflow 

•  Maintenance 
resources 

•  QA 

assignment 


SEiPartn  Agile  Focus  on  Quality  -  Built-In 


1  Executive  Summary 


This  Quality  Assurance  Assessment  Report  documents  the  details  of  an  onsite  Process 
Area  quality  audit.  The  audit  was  performed  by  Software  Quality  from  April  9,  2QJSI  through 
April  15/2009 

|  An  analysis  ot  the  ITD  software  and  project  processes,  documentation,  and  work  products 
was  performed  This  included  review  of  documentation,  overarching  procedures,  Sprint 
retrospectives,  and  bum  down  charts,  notes,  meeting  minutes,  and  interviews  The  analysis 
was  based  on  a  tailored  mini- appraisal  or 'gap  analysis,'  as  described  in  the  CM  Ml  SCAMPI 
B  model 

The  overall  rating  of  this  assessment  indicated  that  the  processes  chosen  to  be  audited 
were  partially-  ,t a  I atggjy- i mplem ented  However,  several  process  areas  were  rated  and  not 
yet  as  they  are  ML 3  and  are  still  in  the  implementation  stage  Assessment  results  will  he 
reported  to  the  associated  project  leaders,  team  leaders,  and  selected  management 


Agile  Resources  to  Develop  Quality  Products 


Assess  ot  Dana  Robttwm 

Updated  for 

Factual  A  ccuiaey  Rebecca  D  icfcen? 


Corrective  actions  and  process  improvements,  based  on  the  assessment  results,  will  be 
documented  and  monitored  A  follow-up  on  the  corrective  actions  derived  from  the 
assessment  results  wvlt  take  place  with  in  three  months  of  acceptance  of  this  report. 


Assessment  Summary  -  J« 

hr  2009 

Pucf  ss  Am 

Ranking 

Measurement  And  Analysts 

9  FI 

Process  and  Ft  o  duct  Quality  Assuiane* 

7  J  U 

Configuration  Management 

NY 

Decision  Analysis  and  Resolution 

9  FI 

Pfoifl  ct  Pluming 

9  FI 

Project  Monitoring  md  Central 

8.4  U 

Integrated  Project  Management 

S  3  LI 

Risk  Management 

SILT 

Requirements  Management 

6  4  LI 

Re  ifuu  e  menlf  D  r velopmenl 

NY 

Technical  Solution? 

NY 

Product  Integration 

mu 

V  eiification  and  Valtdation 

2.BU 

OVER  ALL? 3 LI 

Sprint  Planning/Scrum 


SEI Partner  C*ass  ^crum  Process  ■  Logical  View 


Scrum  Lifecycle  (Iterative) 


■S  Sprint  Planning/Plan 

Determine  sprint  timeline 
Determine  available 
resources-  risks 

^Scrum 

What  done 
What  Planned 
Obstacles  -  Risks 

S  Who,  what,  how 
Requirements 
Coding 
Testing 
Baselines 
Documentation 

S  Deliverable 

Demonstration  to  stake 
holders/CCB 
Deployment  to  User 
. Acceptance.  Testing 
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Sprint  Planning 


Sprint  Planning/Plan 

-  Determine  sprint  timeline 

-  Determine  available  resources  - 
Who,  what  how 

-  Requirements 

-  Coding 

-  Testing 

-  Baselines 

-  Documentation 
Deliverable 

-  Demonstration  to  stake  holders/CCB 

-  Deployment  to  User  Acceptance  Testing 
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Design  Process 


•  Project  Plan 

-  Overall  Design 

•  Sprint  Planning 

•  Daily  Scrum 

-  Developer  review  (Peer) 


Product 

backlog 


Tasks 
to  do 


Completed 
Burndown  tasks 
chart 


SEI  Partner 


Gathering,  Reviewing  and  Testing  the  Requirements 


Enterprise  Architect 


Requirements 
Definition  (Enterprise 
Architect) 


Requirements  Review 
(EA  export  to  web 
interface) 

•  Peer  Review 

•  Developer  review 

•  CCB  review 


Requirements  tied  to 
activities  and  tests  for 
traceability 


SEI  Partner 


Gathering  and  Managing  Requirements  -  BA  &  QA 


~< 


[If  not  Man; 


^1.03  Superv 
and  Ap 


<  ,| _ j 

=:]  Start  Pag  eXK 

Testing 

ft 

T  est 


Manager  Appro 


< 


"  1,0  Request  and  Approval  Process  | 


x 


General  Behavior  Require  Constraints  Links  Scenario  Files  Tagged  Values 
Requirement: 


T  VP 


Difficulty: 


Priority: 


Stability: 


Move  External 


Defined 

Requirement  Type 

1.01-1  Automate  the  entry  of  the  SF-52  (see  data  model).  Functional 

1.01-2  Provide  a  web-based  form  for  entry  of  the  SF-52  accessible  internet.  Functional 

1.01-3  Provide  the  capability  to  track  and  capture  metrics  for  the  SF-52  approval  process.  Functional 

1.01-4  Allow  the  capability  to  capture  authenticated  electronic  or  digital  signatures  for  the  approval  of  the  SF-52.  Functional 

1.01-5  Provide  alerts  when  an  SF-52  submitted  for  approval  has  not  been  approved  within  a  specified  time  frame.  Bus  Rule 

1.01-6  Allow  the  capabiity  to  re-assign  the  SF-52  for  approval  when  the  assigned  Supervisor  or  Manager  is  unavail...  Bus  Rule 

1.01-7  Implement  the  proper  security  and  access  control  for  the  electronic  SF-52  form.  Bus  Rule 


1.01-2  Provid. . . 
1.01-3  Provid. . . 
1.01-4  Allow  t. . . 


Status 
Not  Run 


1 .01-2  Provide  a  web-based  form  for  entry  of  the  SF-52  accessible  by  internet. 


Test: 

Status:  Not  Run 

Run  By:  Dana  Roberson 


[  Description  |  LnpUt  Acceptance  Criteria  I  Results 

3  /  LJ  ^  :=  i=  _ 


Type:  Standard 
Checked  By: 


|  Last  Run:  jo  a 


Build  web  based  SF-52  form  accessubke  by  NNSA  SC,  HQ  and  site  offices. 


sei Partner  Requirements  Management  and  Tracking  -  SE  and  Customer 


D  Assessi  n  g/C  e  rtifyi  n  g/S  e  I  e  cti  n  g 
□  El  eHR  Requirements 

azi  ^Functional:*  1.01-1  Automate  the  entry  of  the  SF-52  (see  data  mo 
IZI  ^Functional:*  1.01-2  Provide  a  web-based  form  for  entry  of  the  SF-i 
IZI  ^Functional:*  1.01-3  Provide  the  capability  to  track  and  capture  me 
IZI  ^Functional:*  1 .01-4  Allow  the  capabilityto  capture  authenticated 
IZI  «Bus  Rule:*  1.01-5  Provide  alerts  when  an  SF-52  submitted  for  apf 
IZI  «Bus  Rule:*  1.01-6  Allow  the  capabiity  to  re-assign  the  SF-52  for  ap 
IZI  «Bus  Rule»  1.01-7  Implement  the  proper  security  and  access  contrt 
IZI  ^Functional:*  1.02-1  Automate  the  entry  of  the  Recruitment  List  (se 
IZI  ^Functional:*  1.02-2  Provide  a  web-based  form  for  entry  of  the  Rec 
szi  «Bus  Rule:*  1.02-3  Implement  the  proper  security  and  access  contrt 
IZI  ^Functional:*  1.03-1  Provide  the  capability  for  the  Approving  Offici 
IZI  «Bus  Rule:*  1.03-2  The  SF-52  must  be  approved  by  a  Approving  O' 
IZI  «Functional»  1.04-1  The  system  should  allow  tthe  requestorto  esta 
fi-^J  «Bus  R i j I e »  1.04-2  Allow  the  Managerto  override  the  Supervisor  a\ 
iid  «Bus  Rule**  1.04-3  The  SF-52  must  follow  an  established  line  of  ap 
EZJ  ^Functional:*  1.05-1  Enforce  business  rule  that  the  Pay  Pool  Admir 
iiiJ  ■*:  F  u  nptfp  n  a  I  **  1.06  -1  Allow  the  SF-52  to  be  submitted  to  HR  when 
tiiJ  «Fun cti onal»  2.1.1  Provide  Notification  or  Queing  Process  for  the  r 
EZJ  ^Functional:*  2.1.2  Allow  Full  Text  Seacrhing  for  Identifying  exisin 
IZI  ^Functional:*  2.1.3  Allow  use  of  existing  PD  template  if  available. 
IZI  ^Functional:*  2.1 .4  Allow  the  entry  of  Position  Description  addendi. 
IZI  ^Functional:*  2.1.5  Allow  Series  to  be  selected  from  a  defined  seH 
IZI  ^Functional:*  2.1.6  Allow  Title  to  be  selected  from  a  defined  selec 
iiil  «Fun cti onab  2.1.7  Allow  Giade  to  be  selected  for  a  defined  select 

U~\  r-  ..  .  „  ...  .  ...... 


iviu,L4i  mcu.  i  mu/iuuo  nvi 

■I'  Project 
•li  Advanced: 


Flow  To 

Flow  From 

Custom  Properties 

Test  Cases 

Other  Links 

Element 

Hama 

Details 

1 .02  Requestor  Fills  out  Recr 

uitment  CheckList 

Activity 

Manader  Request 

Decision 


SEI  Partner 


Coding/Unit  Testing  &  Documentation 


Development 

-  Development  of  Functional  unit  on  DEV  environment 

-  Get  latest  of  all  code,  unit  test 

-  Check  in  Code  to  Visual  Source  Safe 


-  At  prescribed  interval,  build  full  project, 
deploy  to  internal 

-  testing  environment 

-  Automated  Baseline  Capture  developed 
by  team 


•  Well  commented  code 

•  Document  changes  to  other  developers  code 

•  Context  sensitive  help  in  development 
environment  and  customer  environment 

•  Notes  and  resolutions  in  Axosoft  OnTime 


/mm  object;  StoredProcedure  [dbo] , [rsARHearing]  Script 
SET  ANSI_NULLS  ON 
GO  i 

SET  QUOTED_IDENTIFIER  ON 
GO 

E- . 

—  Author:  Shane  Seery 

—  Create  date:  June  2,  2009 

--  Modified  date:  August  11,  2009 

—  Description:  Retrieve  data  for  the  AR  Hearing  Report 
--  added  hearing  request  and  deny  dates  dsr 


BALTER  PROCEDURE  [dbo]  [rsARHearing] 
SStartDate  datetime, 

GEndDate  datetime 
AS 

H BEGIN 

SET  NOCOUNT  ON; 


SEI  Partner 


Developer  Interaction  with  QA  reduces  Risk 


Testing  is  built  into  the  development  cycle 


Early  and  frequent  delivery  ensures  adequate 
time  for  feedback  and  reduces  risk 


Agile  benefits  the  IT  department  assisting  in 
quicker  better  quality  and  reduced  work 


Stressing  repeatable  processes  for  project 
management 


Internal  Testing 
(DEV/QA) 

•  Developer  notifies  QA  of 
functionality  ready  for 
testing 

•  Defect  found  by  QA  - 
entered  into  Axosoft 
Ontime 

•  Developer  fixes  defect 
rebuilds-  workflow  in 
Ontime  Ready  for  testing 
on  TEST 

•  "Defect  Fixed"  or  "Failed 
Retest" 


User  Acceptance 
Testing  (STAGE) 


Projects  Defects 
-  i  User  Dashboard 


Features  Tasks  Incidents  Reports  Tools  Help 


^  Projects 
Projects  Search 
Search  for: 


Project  Wiki  Feature  Backlog  ■  Task  Backlog  |  J  Emails  'v :  Work  Logs  ,  Audit  Trails  +  Incidents  Defect  Backlog 


IS  «  » 


T  -Views  -  jgf)  +  |  V-  y-  IV  “a  Group  -  Sort  -  [£J  - 


-  /- 


Search  for: 


a  a  ^  »>  gi  -  «  i  a  o  j  ¥  j 


Q  t3  All 
El  0 

0 
0 
m  jJ 
0 
0 
El  (J 
El  0 
0 
0 
0 
El  (J 
El  lJ 


Projects 
CATS  SE 
eHR2 
elJP 

ePegasus 

FCDS 

General  Defect  Queue 
ITD 

Maintenance 
Middle  Tier 
OnTime  Setup 
□  TAG 
P3 

PAMS 

0  PAMS  Appraisal  Module 
0  PAMS  Phase  1 
Task  Tracker 
TQP 

VS  Report  Server 
WSU 


Defects  By  Status 

*ft|— 


□ 


Filters  Options 


Status 

Count 

□  pen 

15 

Assigned 

24 

In  Progress 

2 

On  Hold 

3 

Waiting 

2 

Closed 

102 

T  in  Description  (Defect) 


Defect  Id 
2538 
1837 

1310 

1311 
1296 
1237 
1234 
1340 
2529 
2537 
2G71 
1303 
1307 
1303 
1556 
1853 
1B56 
1B57 
2203 
2210 
2213 
2216 
2223 
2383 
1861 
1862 
1863 

1872 

1873 
1875 
1878 
1877 
1873 
1873 
1880 

1832 

1833 
VI 896 

1315 

1316 

1317 
2528 
2531 


Additional  Defect  Info  For  Selected  Item 


■  ^  '■  Work  Log 


Edit  Defect  #1 340 


Average  Vote:  0.00  (0  votes) 
Defect  Information 


Defect  Name 

Rating  Level  on  a  Pert  Log 

Unable  to  do  concurrent  processing  of  two  years  (appraisal  FY09  and  P... 

4.7.1 .1  Pay  Pool  manager  certifies  Pay  Pool 

4. 7.1. 2  After  Pay  Pool  Manager  certifies  paypool  no  changes  can  be  m... 

4.1 4.1 .1  If  employee  is  on  Military  Leave  or  OWCP  the  ratings  bones  in... 

4.1 4.1 .2  If  employee  is  on  Military  Leave  or  OWCP  they  shall  receive  ... 

4.1 0.1 .3  System  must  allow  printout  of  Year  end  review 

Allow  edit  of  assessment 
Supv  to  Subordinate  Supv  Filter 

Establish  a  process  for  closing  out/establishing  an  advisory  rating  and  a  . 

Changing  End  of  Year  Screen 

4.4.1 .1  Reviewing  Official  reviews  recommended  ratings  submitted  in  P... 

4.8.1. 2  Pay  Pool  Manager/Panel  and  can  look  at  assessments 

4. 8. 1.3  Pay  Pool  Manager  /  Panel  can  change  ratings 
Fix  Individual  PayPool  certification  of  excepted  service 
Recovery  Act  on  WORD  v| 

Email  Notifications 
Mid-Year/End  of  Year  weld 
User  can  not  access  PAMEf 
All  data  entered  in  Performs 
Assistance  inputing  an  addi 
System  allows  me  to  enter  £ 

PAMS  access  problem  -  kin| 

When  application  time's  oul 
Supv  SPO  ttl 
Supv  SPO  tt2 
Editing  a  Perf  Plan  after  a  s 
Signature  transfer  to  WORt 
Supv  SPO  in  WORD  Versic 
Refusing  to  Sign  a  Plan 
Auto-filling  WORD  version  t 
Contributing  Factors  in  Perf* 

Assessment  Boxes 
Signing  an  Assessment 
Empl  view  of  Personal  SP0| 

Language  in  Empl/Supv  Si| 

Missing  Part  of  a  SPO 
Single  Sign  On  FWD  reset 
Make  the  sequence  status 
Change  appraisal  status  to 
Make  sure  we  don't  display] 

Time  Out  Alert 
Transposing  SPO's 


Workflow  Step  Priority 

Defect  Verified  Low 

Defect  Ready  for  T esting  on  DEV 
Defect  Ready  for  T esting  on  DEV 
Defect  Ready  for  T esting  on  DEV 
Defect  Ready  for  T esting  on  DEV 
Defect  Ready  for  T esting  on  DEV 
Defect  Fixed  -  Ready  for  Build  on  STAGE 
Defect  Fixed  -  Ready  for  Build  o... 

Defect  Fixed  -  Ready  for  Build  o...  Medi... 
Defect  Fixed  -  Ready  for  Build  on  STAGE  High 
Defect  Fixed  -  Ready  for  Build  o...  High 
Defect  Ready  for  Testing  on  STAGE 
Defect  Ready  for  Testing  on  STAGE 
Defect  Ready  for  Testing  on  STAGE 
Defect  Failed  Retest 


Assi... 

Dan.. 

Aaron. 

Aaron. 

Aaron. 

Aaron. 

Aaron. 

Aaron. 

Dan.. 

Dan.. 

Aaron. 

Dan.. 

Aaron. 

Aaron. 

Aaron. 

Aaron. 


Add  to  Release... 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 

False 


Date  Fixed 


Save  &  Close 


Defect  Name:  [Allow  edit  of  assessment 
Workflow  Step:  Defect  Fixed  -  ReadiJ  for  Build  on  STAGE 
Project:  PAMS  Appraisal  Module 


U  ja. 


Save 


Save  &  New 


Cancel 


0 

6/12/2003 

V'  j 

Build  Number:  1 

!□ 

3/13/2010 

v  | 

Date  Fixed:  □  ■ 

/1 3/2010 

v  1 

Build  Number  Of  Fix:  | 

SRVfiritu- 


"J 


Priority: 

StfltlJS'  A  -^innprl 


a  r 


Details 


1  Notification  List 
,  -]  Description 


Lfa  Related  Items  ||  Customer  Portal  ||  ®  Work  Log  ||  ftpi  Emails 
^  Replication  Procedures  Resolution  ’  Notes 


fi  Alerts  L  History 


%  Attachments 


New  Box — If  a  change  was  made  to  an  employee's  rating  during  the  pay  pool  panel  process  a 
manager  should  be  allowed  to  go  in  and  edit  their  assessment  of  the  employee  to  reflect- justify  the 
new  final  rating. 

Edited  by  Dana  Roberson  on  Friday,  September  03,  2010  at  12:01  PM 

verified  comments  box 


<<  Stamp 
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Acceptance  Testing  and  Traceability 


Name 

Object 

Current 

Status 

Descriptio 

u 

Input 

Acceptance 

Criteria 

PAMS  1.1. 1.2  Supervisor 
can  mits  ate  P  erformance  F 1  an 
for  employee 

1.1.1  Create 
Employee 
Performance 
Plan  -  SPOs 

Not  Run 

If  employeeis 
absent, 

supervisor  can 
create 

performance 
plan  for  them. 

Supervisor  logs  into  PAMS 
Supervisor  selects  Employee 
Supervisor  selects  Plan 
Supervisor  selects  SPO  from 
Library  for  edit  or 

Supervisor  creates  SPO 
Supervisor  saves  SPO 

employee  plan  created 

PAMS  1 . 1 . 1 . 3  An  employee 
can  have  no  more  than  two 
Open  and  Active 

Performance  P  Ians 

1.1.1  Create 
Employee 
Performance 
Plan  -  SPOs 

N  o  t  Run 

Employee  has 
Active  and 
Detail  plan  - 

employee  logs  into  PAMS 
employee  selects  Plan 

Employee  selects  SPO  from 
Library  for  edit  or 

Employee  creates  SPO 
Employee  saves  SPO 

Employee  tries  to  do  this  three 
times 

No  more  than  two 

P  erformance  P  Lan(  S) 
created 

PAMS  1 . 1 . 1 . 5  An  Empl oyee 
can  only  have  two 

Supervisors  (.one  of  Record 
and  one  of  Detail) 

1.1.1  Create 
Employee 
Performance 
Plan  -  SPOs 

No  t  Run 

An  Employee 
may  have  a 
supervisor  of 
record  and  if 
necessary7  a 
Detail 
Supervisor 

Attempt  to  assign  two 
supervisors  to  one  employee 

Can  assign  Detail 

Supervisor,  If  Active 
Performance  Plan  is  in¬ 
activated 

PAMS 

Performance  Plans  with 
reporting  period  of  90  days 
or  more  can  be  evaluated  and 
assessed  for  Final  Rating. 

1.1.1  Create 
Employee 
Performance 
Plan  -  SPOs 

Not  Run 

Performance 
Plans  must 
have  a 
reporting 
period  of  90 
days  or  more 
before 

Attempt  to  rate  Performance 

PLan  of  employee  with  less  than 
90  days  on  job 

System  will  give  warning 
that  employee  has  not 
worked  at  Least  90  days 

=■  Monitoring  and  Controlling  by  Establishing  Measurement 

SEI Partner  0bjectives 


Build  process 
performance 
databases  using 
statistical  process 
control  (SPC). 

>-■  II  — ■  < 


- ! - ^ 

Track  variance 

analyses  (control 
charts  with 
standard 
deviations) 

/ 


Analyze  defects, 
accuracy  of  resource 
estimation, 
development  and 
maintenance 
productivity  and 
customer 
satisfaction 


SEI  Partner 


Measurement  Objectives 


Low-cost  of  high-quality  software  applications  and  services,  delivered 
rapidly,  which  supports  high  levels  of  customer  satisfaction. 

(Mission  Statement) 


IA  =  =-on  Sist“™nt 


%  Budget 

Actual  v  & . 

F  re~De  ployme  nt 

Fost-De  ployme  nt 

Sprint  Burndown 

Maintenance 

Maintenance 

Performance 

Planned  Budget 

DEfect 

Defect 

Variance 

P  rod  Lrcti  v  ity 

Responsiveness 

Performance 

Pe  rforma  nee 

Performance 

--=■  Monitoring  and  Controlling  via  Sprint  Burndown 
SEI Partner  Charts 


%  %  ■  '  ;  :  :  '  %  %  &  :  :  ‘  :  %  %  %  %  { 
B  B  $  B  :j  BBB :  :  :  B  v  B  B  B 


■  ■  •  .  . . . .  . . 


Hours  i.Est: 


Kl 1 1  8§K 


Complete 


Complete 


Complete 


Complete 


Complete 


Burndown  Calc 

115 

Hrs/Day  Baseline 

11.5 

Labor  Hours  Available 

112 

Hours  Not  Used 

-3 

3-Aug 


4 -Aug 


5-Au  g 


S-Aug 


7 -Aug 


34.5 


Reg  ID 


Requirement  Description 


item  Status 


Put  Ax  o  bo  ft  button  in  PAMS 


Complete 


Re w  o  rk  A  p  p  ra  3  a  ■!  statu  b  rep  o  rt 


Pending 


Create  report  notification  for  signature 
status 


Pending 


Make  report  framework  work 


Pending 


1222 -Add  Recovery  Act  checkbox  to 
SPO 


Complete 


1345  -  Make  red  and  green  buttons 
readable  by  the  color  blind 


Pending 


1221  -  Remove  'Check  if  Supervisor  SPO 
is  applicable'  checkbox 


Pending 


1256  -  If  employee  on  Military 
Lea  ve/O  W CP  the  ratings  boxes  greyed 
cut. 


Complete 


Check  and  set  up  interface  items 


Complete 


110 


Rework 


120 


130 


Misc 


Write  :est  Scripts 


PAM 52  Burndown  ( Sprint  1 2) 


27-Jul 


2S-Jul 


29-Jul 


30-Jul 


31-Jul 


115 


140 


Requirements 


■  esting 


71  204345 


103.5 


Sprint 

Velocity 


57.5 


4£ 

%  U'orir 
Completed 


30.5 
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Tracking  Progress  Across  Sprints 


Plan  ned  v  s ,  Actual  Project  Labo  r  Van  ance  {%  J 


Month/ 

Year 

Planned 

project 

Hours 

Actuat 

Hours 

Variance 

% 

iWea/7 

(Avgj 

OCL  LCL 

Maint  % 

Variance 

Mar-08 

2880.00 

2477,60 

13.97 

6.93 

13.32 

0  2,42 

Apr-08 

2250  00 

1745.00 

22.44 

6.93 

13.32 

0 

11.72 

Map-08 

2784.00 

2454  00 

10.73 

6.  S3 

13.32 

0  5.41 

Jun-08 

2596.00 

251750 

3.02 

6.93 

13,32 

0  27.49 

M-08 

3204.00 

2595  00 

13.91 

6.93 

13.32 

0  26.00 

A ug-08 

2047  00 

1  920.00 

6.20 

6.93 

13.32 

0 

12.92 

Sep-03 

1824.70 

1916.00 

5.00 

6.93 

13.32 

0 

16.62 

Oa-08 

2693. 00 

2352  00 

13.41 

6.93 

13,32 

0 

16.49 

Wav-08 

2370.00 

2346.00 

1.01 

6.93 

13.32 

0 

1.04 

Dec-08 

2259,00 

2166.00 

3.26 

6.93 

13.32 

0 

11.24 

J-xk-09 

2583. 00 

2357.00 

8.75 

6,53 

13,32 

0 

13.50 

Feb-09 

2755.00 

2872.00 

4.32 

6.93 

13,32 

0  4,30 

Mar-09 

2321.00 

2258.00 

2.71 

6.53 

13.32 

0 

3.75 

Apr-09 

2857.00 

2542.00 

0.53 

6.93 

13.32 

0 

10.30 

__!£™.riOCi _ 

_ IL-L-VT  _£,j _ £_GJG_ _ 

_ J-5.  _ 

i-i 

^  j  j  j 

Velocity  (Work  Completed/ Sprint  Days) 


30 


OOOOOOOOOOOOOOOOOCOO’- 


>■  : 

y  m 

<  s 


DO  Q-n 
3  ID  “ 

<  CO  ° 


£ 

Q  U- 


s  < ! 


o  o  o 

m  cl  -p;  :=■ 

3  C  23.  o  ^ 

^  <  to  °  Z  □ 


NJonth/Year 


Status  rRYGl 


G 

¥ 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 

G 


Sprint  Mean 

Month/Year  VetocriV  (Avg)  UCL  LCL 

Monthly 

Status 

(RYG) 

Apr-05 

13.32 

14.33 

32.66 

0.00 

G 

May-63 

16.42 

14.33 

32.66 

0.00 

G 

Jun-08 

1 7 .69 

14.33 

32.66 

0.00 

G 

Jvi-03 

10.34 

14.33 

32.66 

0.00 

G 

A  ug-  05 

21 .69 

14.33 

32.66 

0.00 

G 

Sep-05 

22.36 

14.33 

32.66 

0.00 

G 

Qct-QS 

15.53 

14.33 

32.66 

0.00 

G 

S'cv-03 

29.71 

14.33 

32.66 

•0.00 

G 

Dec -05 

24.50 

14.33 

32.66 

0.00 

G 

Fsb-09 

16.30 

14.33 

32,66 

0.00 

G 

.  Max-09 

10.97 

14.33 

32,66 

0.00 

G 
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Maintaining  QA  Assessments 


FOCUS  ARJ 

IA  |  Sol  (write  CoiilJsm.il ion  MniLaEeineut 

Objective: 

Ensure  that  effective  controls  are  in  place  for  the  cooriination  and  iinplementation  of 
changes  to  software  and  its  components.  ' 

ItequDHiuoiftf : 

ShJLimlide: 

A  set  of  activities  for  defe  ct  and  ehhanc  ement  identific  ation,  prica±mtion  and 
selection;  document  and  code  version  tracking  and  control;  and  software  installation 
track;  ins. 

Apprtidi 

Determine  the  existence  -of  a  Software  Configuration  Management  Plan  (SCMP), 
either  a s  a  standalone  document  or  embedded  in  another  document,  and  ensure  that  il 
specifies,  as  $pplicable,the  following: 

*  Change  control 

*  Version  control 

#-  Configuration  item  idoitification  and  classify  ation 

*  Issues  management 

tnteni: 

lUnlt  ; 

1, 

All  software  ccorpona its  aid  products  to  be  managed  have  been  identified, 

9 

2. 

Authority  for  change  is  defined. 

9 

3. 

Baselines  for  all  identified  walk  products  are  established  and  the  integ±y  is  maintained. 

9 

4. 

Changes  to  work  products  are  tracked  and  controlled. 

9 

5. 

Process  for  coating,  tracking,  and  resolving  issues  is  defined. 

9 

\r 

Changes  to  the  source  code  can  be  linked  to  an  associated  issue. 

9 

7. 

Issues  associated  with  the  software  product  are  tracked  and  maintained. 

9 

E. 

Releases  of  the  software  product  are  tracked  and  maintained  and  a  listing  of  the  different 
versions  is  available  such  that  a  previous  release  of  the  product  can  be  made  available 

9 

9. 

Processes  are  followed. 

9 

IE. 

Periodic  canfigumion  assessments  and  reviews  are  conducted  and  documented. 

9 

SCORE: 

4  (FI) 

Tukxmg: 

Discu isj  here  tailoring  the  assessmeti  was  rilfaitedih  tUsfocas  area  ft 

application  im&r  review. 

7T  the 

RHoidi 

RflOflved: 

List  di  tiie  records  that  were  reviewed 

VSS  histoay 

Project  Log 

CMP 

Sprint  Backlog 

Inkvuiw 

Conducted: 

list  di  interviews  that  were  conducted  r'ti1  nd  include  the  n&ttes  cf  individuals,  tui 

instead  titles) l//- . — - . 

Management 

Developers 

Executive  Summary 


This-  Quality  Assurance  Assessment  Report  documents  the  details  of  al 
Area  quality  audit.  The  audit  was  performed  by  Software  Quality  from  April  a,  ^ 
Aprill  5 ,2009 


An  analysis  of  the  ITD  software  and  project  processes,  documentation,  and  work  products 
was  performed,  This  included  review  of  documentation,  overarching  procedures,  Sprint 
retrospectives  and  burn  down  charts,  notes,  meeting  minutes,  and  interviews.  The  analysis 
was  based  on  a  tailored  mini-appraisal  or  "gap  analysis,"  as  described  in  the  CMMI  SCAMPI 
B  model. 


The  overall  rating  of  this  assessment  indicated  that  the  processes  chosen  to  be  audited 
were  However,  several  process  areas  were  rated  arid  not 

yet  as  they  are  ML  3  and  are  still  in  the  implementation  stage  Assessment  results  will  be 
reported  to  the  associated  project  leaders,  team  leaders,  and  selected  management. 


Assessment  Summary- Ju 

lv  2009 

Process  Area 

Ranking 

Measurement  and  Analysis 

9  FI 

Pr o  c  e  s  s  and  Pro  duct  Quality  A  s  surance 

7.5  LI 

Configuration  Management 

NY 

Decision  Analysis  and  Resolution 

9  FI 

Project  Planning 

9  FI 

Proj  e  ct  M  onitoring  and  C  ontrol 

3.4  LI 

Inte  grate  dProjectManagement 

3.3  LI 

Risk  Management 

3.2  LI 

Requirements  Management 

6.4  LI 

Requirements  Development 

NY 

jr 

Technical  Solutions 

NY 

Product  Integration 

3.6  LI 

V  er if ic ation  and  Validation 

3.3  LI 

OVER  ALL  3.3  LI 

Assessor: 
Updated  for 
Factual  Accuracy: 


Dana  Roberson 


Rebecca  Dickens 


Corrective  actions  and  process  improvements,  based  on  the  assessment  results,  will  be 
documented  and  monitored  A  follow-up  on  the  corrective  actions  derived  from  the 
assessment  results  will  take  place  within  three  months  of  acceptance  of  this  report. 


SEiPartner  Tracking  Quality  Corrective  Actions 


0 


Process  Area 


# 

artifacts 

reviewed  Findings 


ft  FI  #  LI  #  PI  ft  NY  ft  CAP 


ft  CA 

completed  Date  Completed 


Re-Evaluation 

Date 


Organizational  .raining 


13 


1  Need  to  shew  where  relevant 
stakeholders  have  teen  involved 
training  in  OT 

2  Need  to  show  upper  management 
and  get  responses 


Organizational  Process 
Definition 

3 

1 

1 

1  Need  to  show  upper  management 
and  get  responses 

Organizational  Process 
Focus 

3 

1 

1 

1  Need  to  show  upper  management 
and  get  responses 

Measurement  and  Analysis 

5 

1 

1 

1  Nsed  to  show  upper  management 
and  get  responses 

Process  and  Product 

Quality  Assurance 

9 

2 

1 

1  Need  to  show  upper  management 
and  get  responses 

2  Need  to  have  objective  evaluation  of 
all  in  house  evaluations 

Configuration  Management 

4/21/2009  OT- Training  held  on  11/12/OS 
for  the  full  team  Emails  show  training 
announcement  and  follow-up  with  Frank. 

Surreys  however  have  been  lost  due  to  change 
in  evaluation  database  (This  applies  to  OPD  and 
OPF  as  well  as  they  were  all  handled  together ! 


Jun-09 

Jun-09 

Jun-09 

Jun-09 


~^r  Lessons  Learned  in  Managing  Risks 
SEI Partner  Usjng  CMM|.DEV  and  Agile 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Management  of  complexity  requires 
process  discipline  WHILE 

Management  of  change  requires  rapid 
adaptability. 


CMMI  provides  process  discipline. 


SCRUM  (Agile)  enhances  adaptability 
and  commitment. 


~^r  Lessons  Learned  in  Managing  Risks 
SEI Partner  Usjng  CMM|.DEV  and  Agile 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


Understand  that  all  process  improvement  opportunities,  as 
with  all  investments,  have  costs  and  risks.  Introducing  Agile, 
as  with  any  new  technology,  needs  the  advocacy  of  all  project 
stakeholders. 


Participation  of  key  staff  members  in  planning  practice 
implementation  is  needed  to  understand  the  continuity  and 
costs  and  identify  the  key  risks,  such  as  customer  culture. 


The  CMMI  Risk  Management  Process  Area  practices  are 
easily  tailored  and  implemented  via  a  suitable  template  for 
Agile  based  projects. 


The  Agile  method  emphasizes  on-going  requirements  & 
design  verification,  daily  SCRUMS  with  customer 
participation  and  team  “espirit  de  corp”.  The  Agile  emphasis 
on  these  activities  provides  engineering  practices  that  reduce 
risks  in  software  engineering  tasks. 
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Remember 


SEI  Partner 


CMMI-DEV  is  a 
process  model, 
NOT  a  process 
description. 


CMMI-DEV  only 
defines  “WHAT” 
to  do,  not 
“HOW”  to  do  it. 


Plan  the  process 
implementation 
so  that  you  are 
taking  advantage 
of  the  CMMI-DEV 
practices  AND 


Use  the  Agile 
Methodology  for 
rapid  turn  around 
for  low  cost 
quality  product. 


Copyright  ©  2C05,  Mountain  Goat  Software 


SEI  Partner 


The  End 


Transdyne  Corporation 
http://transdvnecorp.com 

CMMI  Implementations  in 
Small  &  Medium  Organizations 


You  have  just  seen  key  benefits  of 
using  CMMI-DEV  and  Agile  to  reduce 
risks  in  software  engineering  from  the 
“30,000  feet”  level. 


Questions  or  Comments  ? 
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Examples  of  Mapped 
CMMI  Specific  Practices 
to  Agile  Life  Cycle  Steps 


Artifacts 


1.  Agile  Life  Cycle  Start 
Project  Pre-Work  —  Prior  to  initial  iteration 


Share  Point  M-ode 
Project  Repository 


1  O  Develop  l^se  Case 
Model  including  Use  Case 
Point  cahijjla:  on  to  s  upper  £ 
effort  estimation 


CMMI  SPs 


Enterprise  Architect 
Package 
Requirements  and  Design 
tool 


PP  2.5 
PP  3.1 
RP3.2 
PP  3.3 
REG-M  1 .2 
MA  SG  1 


PP  1.1 
PP  2.1 
PP2.3; 
PP  2.4 


i£ 


1.1  Complete  and 
present  Project  Overview 
to  ITRB 


REQM  1 . 1 
REGM  1.4 
PP  1 . 2 
PS  KM  2.1 


35. 


CMMI  SPs 


12  Establish  Project's- PAL 


PP  1.2 
PP  1.3 
PP  1.4 
PP  2.4 
PP  2.5 
PP  2.6 


22Z 


v 


1 .3  Establish  Project's 
requirements  and 
design  Structure  in  EA 


REQM  1.5: 
CM  1.2; 
CM  1  3: 
QPD  1 .5; 
MA  1.3: 
PP  2.5 


1.5 


Define  Project  timeline, 
scope,  goals,  and 
objectives,  stakeholders 
(IPTJ  srid  major  del.veraWes 


3£ 


/L 

Nr 


1  4.  Dev  high-level 
understanding  of 
software  business 
ccntext- 


REQM  1.3: 
RD  3.2 


AqqliH  Artifact 

PP  1.2 

PP  2. 1 

1  7  Develop  Initial  WDS, 

PrcijjacL  Bacldog  including 

PP  2.2 

estimates  ard  iteration  and 

hours  and  estimates 

PP  2.4 

release  plans 

PP  2.2 

PP  2.7 

PP  2.6 


Ik 


1.8  Plan,  facslita  u&  and 
document  initial  stakeholder 
meeting 


-js 

-v 


1  y  Obtain  approval  to 
proceed  la  natft  iteration 


it 


REQM  1.2 
PP  3.3 


1.10  Prepare  and 
distribute  initial  status 
report 

Project  StaLus  RreporL 

Artifacts 


1 .  Agile  Life  Cycle  2.0 
Initial  Iteration 


Enterprise  Architect 
Package 

Requirements  an d  Design 
tool 

SRS  Content 


Agile  Artifact 
Project  Backlog 


Agile  Artifact 
Project  Backlog! 


Projtfri^  R report 


CMMI  SPs 


REOM1.3 
VER  5.1 
VER  5.2 
VER  5.3 


PMC  SG  2 
FP2.4 
PP5.6 
PP2.7 


REQM  1 .3 
REQM  1.3 
PP  1.1 
PP  1.5 
PP  5.5 


pp  3.e 


1.10  Prepare  and  distribute 
Status  Report 


CMMI  SPs 

REQM  1 . 1 
REQM  1.2 

Ft 

RD  2.1 
RD  1.1 
RD  1.2 


PMiC2.1 
PMC  2.2 


REQM  1  3: 
RD  3.2 


PMK2  1.6 
PMC  1 .7 
GP  2_1Q 


Lean  Development  - 
Influencing  the 
Organization 
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"  Introduction 

■  Leverage  Points 

■  System  View 

"  Description  of  the  different  type  of  leverage  points 

■  How  to  apply  to  your  project 

■  But  first  -  a  story 

■  Electronic  company  with  good  metrics 

■  Root  cause  performed  but  not  effective 

■  Declining  sales  and  quality 

■  Program  meetings  focused  on  delivery  and  shipping  metrics 
-  Quality  and  production  metrics  always  last  ...if  at  all 

■  End  of  one  month,  workers  gathered  the  wheel  stops  from  the  parking 
lot!?? 
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The  Introduction  -  Leverage  Points 
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■  Leverage  points  are  places  within  a  complex  system 
where  a  small  shift  can  cause  big  changes 

■  Best  describe  by  Donnella  H.  Meadows  - 12  leverage  points 

■  Key  to  influencing  the  organization 

■  Leverage  points  are  not  a  new  idea  -  imbedded  in 
history,  culture,  and  Legend 

■  Intuitively  aware  -  used  wrong 

■  Most  organizations  know  about  their  leverage  points 

-  A  given  improvement  team  may  not 

■  Levers  are  often  counterintuitive  -  pushed  when  they  should  be  pulled1 

-  Strongest  saving  often  come  from  redirecting  the  organizational  force  on 
their  levers 

1  J.W  Forrester,  World  Dynamics  Portland  Oreg.:  Productivity  Press  1971. 
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Introduction  -  Leverage  Points 
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■Traditional  view  of  the  forces  acting  for  and  against  any 
improvement  process 

■  Budgetary 

■  Organizational 

■  Social 

■  Strategic 

■  System  thinking  helps  to  rank  and  categorize  these  forces 

■  Six  Sigma  “SIPOC”  is  a  starting  point 


Copyright© 2010  Boeing.  All  rights  reserved. 


Author,  11/22/2010,  Filename. ppt  |  4 


The  Introduction  -  A  System  View 
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■  Process 


Process 


Process 

Measures 


Discrepancy 


Process 

Expectation 


Output 

feedback 

loop 


■  Inputs  increase  the  available 
stock  needed  for  the  process 

■  Outputs  decrease  the  available 
stock  created  by  the  process 

■  The  Process  Measures  VS  the 
Process  expectation(s)  the 
discrepancy 

■  Two  negative  feedback  loops 
adjusts  the  input/output  (WIP 
Control) 

■  The  Process  expectation  and 
the  feedback  loops  are  outside 
the  process 
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Introduction  to  the  Levers 
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■Some  levers  are  stronger  than  others 

■  The  order  of  strength  of  the  levers  may  vary 

■  Resistance  to  change  higher  for  stronger  levers 

■  Levers  take  various  forms  (lowest  to  highest) 

■  Physical  -  The  “Hows” 

■  Response  -  The  “Whats” 

■  Transformational  -  The  “Whys” 
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Physical  -  The  lowest-level  Levers  (1  of  2) 
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■  Physical  -  lowest-level  Levers 

■  12.  Numbers 

-Think  metrics,  staffing  levels,  number  of  Cad  terminal,  etc 
■11.  Buffers 

-Think  Overtime,  work  constraints,  WIP,  material  stocks,  etc 
-Has  to  be  thought  about  in  terms  of  the  process  needs 
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Physical  -  The  lowest-level  Levers  (2  of  2) 
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■  Physical  -  Lowest-level  Levers 

■  10.  Process  structure  or  network 

-Think  hiring,  office  space,  ability  to  add  CAD  terminals,  machines, 
etc. 

-  Machining  centers,  transfer  lines,  computer  networks 

■  9.  Delays  relative  to  the  rate  of  changes 

-Think  ability  to  respond:  audit  results,  metrics,  reaction  to  standard 
and  special  causes,  etc 

-How  fast  can  you  gear-up?  wind-down? 

-How  fast 

-What  happens  if  a  very  large  order  is  received? 

-How  fast  can  your  systems  respond? 
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Response  -  The  Mid-level  Levers 
Response  (1  of  2) 
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"  Response  -  Mid-level  Levers 

■  8.  Negative  feedback  loops 

-  Think  about  how  people  respond  to  metrics,  requests,  issues,  risks,  etc 

-  Does  it  change  based  on  who  is  involved? 

-  Slows  the  process 

■  7.  Positive  feedback  loops 

-  Think  awards,  encouragement,  etc. 

-  Often  hard  to  quantify  -  maybe  intangible  and  invisible  until  the  effect  are 
felt 

-  Speeds  the  process 
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Response  -  The  Mid-level  Levers  (2  of  2) 
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■Response  -  Mid-level  Levers 

■  6.  Information  transparency 

-Who  controls  the  information  and  who  can  see  the  information 

-Think  about  what  change  boards  or  other  forums  the  data  must  go 
through 

-Who  are  the  information  gatekeeper  in  your  organization? 

-This  is  why  a  given  improvement  team  may  not  know  the 
organizational  leverage  points 

■  5.  Rules 

-Think  about  the  command  media  system 
-  Process  Governance 
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Transformational  -The  High-level  Levers 
(1  of  2) 
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■Transformational  -The  High-level  Levers 

■  4.  The  power  of  self-organization 

-Direct  impact  on  Process  Discipline 

-Group  theory:  Tuchman’s  “forming  storming,  norming,  performing” 
-Organizational  Framework:  Zachman  -  layers  are  self-organizing 

■  3.  Process  expectations 

-Think  about  how  the  expectations  impacts  the  feedback 
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Transformational  -The  High-level  Levers 
(2  of  2) 
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■Transformational  -The  High-level  Levers 

■  2.  Process  mindset  or  paradigm 

-Think  about  the  underlying  values  of  the  organization,  management, 
corporate  culture,  etc. 

-The  lever  is  not  easily  move  for  a  specific  improvement  process. 

-May  determine  what  type  of  solutions  are  acceptable 
-Six  Sigma’s  “waves”  approach 

■  1.  Belief 

-Strongly  held  belief 

-  Personality  testing  and  surveys  hints  at  how  this  is  working  in  the 
organization 

-The  organization  promotes  some  while  down  playing  others 
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Leverage  Points  -  How  to  apply 
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■  Look  beyond  just  the  facts 

■  When  performing  the  analysis,  understand  what  levers  are  influencing 
the  project 

-  E.g.  Value  stream  maps  often  only  describe  the  “Physical”  and  some 
“Response”  levers  -  Add  the  other  “Response”  and  “Transformational” 
levers 


■  When  looking  for  solution(s),  address  multiple  leverage  points 

-  E.g.  A  new  metric  is  a  “Physical”  lever,  changing  the  documented  process 
is  a  “Response”  lever,  Shifting  the  goal  of  the  process  is  a 
“transformational”  lever. 

-  Sometimes  just  adjusting  the  way  a  lever  is  applied  is  the  best  solution. 
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Leverage  Points  -  Conclusions 
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■The  organization  knows  their  leverage  points 

■  A  given  improvement  team  may  not 

■  Most  organizations  are  using  them  wrong  -  counterintuitive 

■  Promote  multiple  leverage  solutions  with  the  same  goal  for 
maximum  impact 

■  Ensure  the  solution  is  capture  in  the  processes/systems  for  long  term 
savings 

■  Work  towards  transformational  changes 
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The  rest  of  the  Story 
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■  Shipping  the  wheel  stops  to  various  customer  because  they 
we  short  on  a  key  metric:  “Tons  Shipped.” 


■  Only  metric  the  CEO  was  really  concerned  about 

■  Prior  job  a  CEO  of  a  steel  company 

■  Solution  required  changing  the  expectation  of  the  processes 

■  Only  then  does  the  quality  and  production  methods  show 
improvements 
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■  Donella  Meadows,  Leverage  Points:  Places  to  Intervene  in  a 
System ,  1 999 

http://www.sustainabilitvinstitute.org/pubs/Leveraqe  Points. 

pdf 
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■  Introduction  to  tools  and  the  usage 

■  Introduction  to  Enterprise  Frameworks 

■  Framework  -  Why  should  I  care? 

■  Description  of  aFramework 

■  Description  of  the  different  layers  of  the  framework  and  their 
role  in  improvement  efforts 

■  How  to  apply  to  your  project 
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Introduction  to  tools  and  the  usage  -  CMMI 
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■  See  how  I  do  my  tasks  by  writing  them  -  defined 
processes 

■  Am  I  following  my  defined  processes? 

■  Am  I  measuring  my  defined  processes? 


Copyright© 2010  Boeing.  All  rights  reserved. 


I  can  identify  improvements  because 
they’re  defined,  controlled  and 
measured 
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Introduction  to  Tools  -  Tactical  solutions 
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Process  improvement 
methods,  models  and 
techniques  are  like  tools. 

■  Pick  the  tool  for  the  job 

■  If  you  only  have  a  hammer  in 
your  tool  box,  every  problem 
will  look  like  a  nail! 

■  Tools  are  used  together  to 
insure  overall  improvement 


Tool  Box 

Six 

Sigma 

Lean 

Theory  of 
constraints 

Theory 

Reduce 

variation 

Remove 

waste 

Manage 

constraints 

Focus 

Problem 

focused 

Flow 

focused 

Systems 

constraints 
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Introduction  to  Tools  -  Six  Sigma,  Lean, 
TOC  Comparison 
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■  Complexity  of  problem 

■  High  -  Theory  of  Constraints 

■  Medium  -  Lean 

■  Low  -  Six  Sigma 

■  Organizational  culture 

■  Authority  driven  -  Theory  of  constraints 

■  Authority  approved/team-shared  work  -  Six  Sigma 

■  Teamwork  directed  -  Lean 

■  Driving  factors 

■  Logic  -  Theory  of  Constraints 

■  Data  -  Six  Sigma 

■  Flow  -  Lean 

■  ROI  visibility 

■  High  -  Six  Sigma 

■  Medium  -  Theory  of  Constraints 

■  Low  -  Lean 
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Introduction  -  Frameworks 
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■The  biggest  Issue  with  improvement  tools  is  not 
learning  the  tools  but  proper  application. 

■  Improvement  methods  widely  defused 

■  Shotgun  approach  ineffective 

■  Organizations  have  hard  time  focusing  efforts 

■  Low  hanging  fruit  picked 

-  Need  to  make  improvements  higher  in  the  organization 

-  Need  to  improve  the  organization’s  health 

■  Frameworks  are  key  to  focusing  the  organizational 
efforts 


Copyright© 2010  Boeing.  All  rights  reserved. 


Author,  1 1/22/2010,  Filename. ppt  |  6 


Introduction  -  Frameworks 
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■Most  people  have  some  understanding  of  the 
organizational  framework. 

■  Organizational  chart 

-  I  know  my  boss/program/division/company/etc 

■  Learned  in  High  School  English  the  5  W  and  H. 

-  What,  Where,  Who,  When,  Why  and  How 

■  Understand  the  organization  has  different  functions 

-  Executive  leaders,  IT,  Engineering,  Quality,  Operations,  etc 

-  May  or  may  not  understand  their  respective  relationship 

■  A  Framework  is  simply  helps  to  integration  of  things 
and  their  relationships 
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■The  layers  of  the  framework  are  self-organizing 

■  Will  always  work  to  be  integrated 

■  Will  not  inherently  be  efficient  or  effective 

■Resolving  conflicts  in  the  frameworks  are  key 
to  process  improvement 

■  There  will  always  be  conflict  in  the  framework 

■  Unresolved  conflict  will  increase  uncertainty  resulting  in 
increased  cost  and  schedule  while  decreasing  quality 

■  Conflicts  often  are  resolved  in  ways  that  create  constraints 
as  the  organization  changes 


Copyright© 2010  Boeing.  All  rights  reserved. 


Author,  1 1/22/2010,  Filename. ppt  |  8 


Framework  -  Why  should  I  care? 
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■The  layers  of  the  framework  are  self-organizing 

■  Will  always  work  to  be  integrated 

■  Will  not  inherently  be  efficient  or  effective 

■Resolving  conflicts  in  the  frameworks  are  key 
to  process  improvement 

■  There  will  always  be  conflict  in  the  framework 

■  Unresolved  conflict  will  increase  uncertainty  resulting  in 
increased  cost  and  schedule  while  decreasing  quality 

■  Conflicts  often  are  resolved  in  ways  that  create  constraints 
as  the  organization  changes 
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Introduction  -  Frameworks 
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■Classification  system  is  by  John  A. 
Zachman 

-6X6  Matrix 

-5  W  and  H  across  the  top 

-6  different  “transformational”  layers 

-Available  from  http://www.zachmaninternational.com/ 

■DODAF  2.0 
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The  easiest  savings  comes 
from  resolving  conflicts 
within  a  layer. 


What 


"  Resolving  conflicts  between 
two  layer  results  in  10X  the 
savings 


■  The  higher  the  conflict  is  in 
the  architecture  the  greater 
the  saving  and  the 
resistance 

■  The  integrating  a  column 
multiples  the  savings  as  the 
conflicts  between  all  the 
layer  of  that  column  are 
resolved. 
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Primary  activity  of  the  layer 
change  and  thus  need  to 
use  different  tools 


What 


Top  two  layer  are 
addressing  the  “Whys”  of 
the  business 

The  middle  two  layer  are 
addressing  the  “Whats”  of 
the  business 

The  bottom  two  layer  are 
addressing  the  “Hows”  of 
the  business 

Tools  need  to  match  the 
primary  task  of  the  layer 
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Frameworks  and  Tool  Selection  -  How  to 
apply 
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■Attempt  to  fill  in  the  framework 

■Difficulties  in  completing  highlights 
improvement  opportunities 

■Empty  blocks  or  controversial  block 
answers  highlight  conflicts 

-Upper  layer  use  Theory  of  Constraint 

-Middle  level  use  Design  for  Six  Sigma 

-Lower  levels  use  Lean/six  sigma 
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Frameworks  and  Tool  Selection  -  How  to 
apply 
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■Multiple  blocks  within  a  layer 

■For  mid  or  lower  level  layer  -  Determine  if 
issue  is  flow  or  data  or  conflict 

-Flow  -  Lean 

-Data  -  Six  sigma 

-Conflict  -  Theory  of  Constraints 

■For  high  level  layer  start  with  Theory  of 
Constraints 
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Frameworks  and  Tool  Selection  -  How  to 
apply 
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■Vertically-aligned  blocks  are  conflicts  and 
transformations  difficulties 

■Theory  of  Constraints  preferred  tool 

■  Focus  effort  to  resolve  similar  business  questions 
■Caution  not  to  jump  layers 

■  For  high  level  layer  start  with  Theory  of 
Constraints 
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Conclusions 
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■There  are  multiple  tool  boxes  with  a  wide  range 
of  tools 

■This  presentation  only  touched  on  Three 

■Using  a  framework  helps  to  find  improvement 
opportunities 

■  Holes  in  the  framework  highlight  the 
organizational  conflicts 
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Conclusions 
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■Resolving  conflicts  in  the  architecture  results 
in  saving. 

■  Higher  in  the  framework  the  greater  the  saving 

■Conflicts  between  the  layer  result  in  larger 
savings  than  resolving  the  conflict  within  the  layer 

■Knowing  the  conflict  type  and  location  helps 
determine  the  correct  tool  set. 
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■Zachman  information 
http://www.zachmaninternational.com/ 

■DODAF  2.0  information 
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CMMI  Implementation 
Strategies  for  Success 

Stephen  M.  Austin 


LOCKHEED  MARTIN 


7 


Akron,  OH 


MS2  Locations 


Syracuse,  NY 


Eagan,  MN 


Manassas,  VA 


Baltimore,  MD 


Geographically  Dispersed  -  13,000  Employees  in  6  Primary  Locations 


2 


MS2  Path  to  CMMI  Level  5 


MS2  Measurement  Prog 
Core  Components 


Manage  Strategy 
J  Goals  and 
{objectives 


Closed-Loop  Management  Processes 


Decision-Maker 
&  Stakeholder 
Feedback 


Plan 


Act 


Measure 


Control 


Assess  & 
Control 
Performance 


Define  Measurement 
Resources,  Goals  & 
Information  Needs 


Establish  &  Plan 
Organizational 
&  Program 
Measurement 
Activities 


Prioritized 
Info  Needs 


Candidate 

Measures 


MS2  Organizational  Standard 
Process  &  Program 
Management  Plan  (PMP) 


Select  &  Specify 

1  Measures 

Measures 

J 

Collect,  Validate 
&  Store 

Processed 

Data 

Measurement 

Data 

Organizational  & 
Program 
*  Measurement 
Plans 


Configuration 
Control  of 
Measurement 
Work  Products 


) 


Analyze  Measures 


Perform  Org 
and  Program 
Measurement 
Activities 


TQu« 


Integrated 
Analysis  Results 
&  Performance 
Measures 


Measurement 
Results  &  Take 
Action 


Improvement 

Actions 


Evaluate  and  Sustain  Organizational  Measurement  Assets 


To  Decision- 
Makers  & 
Stakeholders 


Measurement  Activities  Information 

•  Measurement  Plan 

•  Analysis  Results 

•  Performance  Measures 


Common 

Measures 


Defined  Process 


Business  Process  Management 


Organizational  Process  Measure 


i  TiMonng  Cycle  1 
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MS2  Functions 


Project  1 
Project  2 


Project  N 


Discrete  Goals  Allocated  to 
Lines  of  Business  (LoBs), 
Functions  &  Programs 


Measurement 

Repositories 


Historical 

Performance 


Lockheed  Miirtin  MS2 
room  in  Measurement  Plan  Template 


Goal  Flowdown  Drives  Information  Needs 
Across  the  Enterprise 


Final  Thoughts... 


•  Once  you  make  the  Decision  to  Pursue  an  Appraisal  -  Run  it 
Like  a  Program 

•  Centralize  Activities  that  Need  to  be  Common 

-  Measurement  Infrastructure  Supports  both  the  Organization 
and  Programs 

-  Robust  Organizational  Measurement  Repository 

-  Establish  set  of  PPBs  and  PPMs  to  meet  Business  Needs 

-  PPMs  satisfy  intent  of  “Healthy  Characteristics” 

-  Continue  to  refine  /  evolve  PPBs 

•  Leverage  existing  efforts  as  much  as  possible: 

-  Programs  leverage  Black  Belts  for  statistical  thinking 

-  Existing  Operating  Excellence  efforts  synergistic  w /  Level  5 
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Improve  Patient  Care. 
Maximize  Business  Results. 


Implementing  CMMI  in  a 
Virtual  Development 
Organization 

Rick  Bennett,  Director  of 

Development 


November  16,  2010 

Med 


IIS» 

A  Quest  Diagnostics  Company 


MedPlus 


■  Customer  facing  software  division  of  Quest 
Diagnostics  Incorporated. 

>  Hospital  /  IDN 

>  HIE  /  Government 

>  Physician  Office 

■  Quest  Diagnostics  is  the  world’s  largest  chain  of 
clinical  diagnostic  laboratories. 

>  Grown  by  acquisition 

>  Disparate  systems  and  data  centers 

>  Greater  than  a  half  million  specimens  per  day 


A  tkiEsst  Diagnostics  Company 


Care  360  Lab  Orders  and  Results 


■  Originally  developed  in  2000 

■  >60  %  percent  of  Quest  Diagnostics  orders  pass 
through  Care  360  LOR 

>  300,000  per  day 

■  990,000  registered  users 

■  44,000  simultaneous  peak  users  per  day 

■  42,000,000,000  unique  results 
■19  TB  of  data 

■  Six  Sigma  up  time 

■  Every  specimen  represents  a  human  life 

■  Awarded  United  States  Patent  #7,567,913 

MedPlusw 

A  fliiDst  Diagnostics  Company 


Diagnostics  Services  Group 

■  Extremely  virtual  team 

■  45  Developers,  requirements  analysts,  QA  analysts,  support 
analysts,  PM’s  and  resource  managers  in  16  locations 

■  Modified  waterfall  SDLC 

■  4  releases  per  year 

■  Six  Sigma  trained 

>  2  Black  Belts,  >10  Green  Belts 

■  Daily  all  hands  calls 

■  Heavy  use  of  instant  messaging  and  smaller  conference  calls. 

■  High  employee  satisfaction 

■  Function  points  analysis  showing  consistently  high 
productivity 

A  fliiDst  Diagnostics  Company 
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A  fliiDst  Diagnostics  Lomnary 


Why  CMMI? 

■  Low  CMMI  penetration  in  Healthcare  IT 

>  Not  a  necessity  for  sales  in  our  market  space 

■  Create  a  more  predictable  process 

>  Make  post  release  week  feel  like  any  other  week 

■  Improve  quality  where  possible 

■  Align  to  customer  goals 

■  Maintain  autonomy 

>  Team  structure  and  geography  unique  to  our  organization 


A  tkiEsst  Diagnostics  Company 


How  did  we  get  to  Level  5? 

■  Management  involvement 

>  Training 

■  Leveraged  Six  Sigma  training  and  culture 

>  Access  to  Master  Black  Belts 

■  Stretch  to  achieve  a  level  then  refine  those  processes 
while  pushing  to  achieve  the  next  level. 

>  SEPG 

>  Obtain  practitioner  feedback  and  refine 

>  Perfect  is  the  enemy  of  good  for  initial  processes 

■  Keep  focus  on  the  reasons  for  process  improvement 

■Tap  into  innovative  spirit 

>  Particularly  at  the  High  Maturity  levels 

MedPlusw 

A  fliiDst  Diagnostics  Company 


Results 


■  Assessed  at  CMMI  Level  2  -  August  2005 

■  Assessed  at  CMMI  Level  3  -  July  2008 

■  Assessed  at  CMMI  Level  5  -  June  2010 

■  Acknowledged  process  leaders  in  our  organization 
and  company 

■  Increased  visibility  into  status  of  processes  during 
development  phases  and  increased  understanding  of 
process  capability  yielding  increased  confidence  in 
final  product 

■  Increased  visibility  into  linkage  between  micro 
activity  and  fulfillment  of  customer  goals 


A  tkiEsst  Diagnostics  Company 


QPPO  Dashboard. 


E^l 


0  https  //clarity  qdx.com/niku/app?action=dms  viewFileftfileld  =  1 1451G58ftfileName=CMMI  Dash  boa  id  20  -  Windows  Internet  Explorer  pro 
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Questions? 


A  {kiEsst  Diagnostics  Company 


Rick.J.Bennett@questdiagnostics.com 


CMMI®  Implementation 
Strategies  for  Success  Panel 

16  Nov  2010 


Discussion  Topics 


Strategies  for  Success 

How  to  implement  good,  effective 
measures  to  determine  what  CMMI 
specifically  does  for  your  organization 


Benefits  of  CMMI 


Steps  to  Success  Using  CMMI 

Understand  Your  Organizational  Objectives 

•  Just  to  get  organized?  .  |mprove  Quality? 

•  Get  an  External  Ratin9?  .  Delight  Customers? 

•  Reduce  Costs? 

Ensure  top-down  buy  for  Objectives  &  Strategies  to  achieve 

•  “CMMI”  objectives  should  not  be  standalone 

Leverage  Industry  Experience 

•  Reasonable  expectations  for  velocity  of  change 

•  Benchmark  cost,  schedule,  tactical  plans 

Establish  Measures  for  Success 

•  Meaningful  to  the  organization 

•  Consider  items  that  measure  achievement  over  time 

•  Get  specific 


Measures  of  CMMI  Effectiveness 


For  Organizations  just  starting  out. . . 

•  %  of  target  employee  group  that  knows  where  to  find 
processes  and  how  to  suggest  changes  (survey) 

•  Count  of  “escapes”  attributable  to  poor  or  unknown 
process  instruction 

•  Specific  process  area  metrics 
For  More  Mature  Organizations  ... 

•  Measures  of  execution  improvement  (overall  or 
targeted  improvement  initiatives) 

•  Cycle  time 

•  Quality 

•  Technical  performance 

•  customer  satisfaction 

•  Extent  of  process  improvement  involvement  from  the 
organization  (#  suggested,  %  implemented,  analysis  of 
origin) 


Benefits  of  CMMI 


Provides  a  neutral  framework  for  best  practices 

•  No  need  to  invent/maintain  your  own  standard 

•  Common  language  across  diverse  business  &  customer  base 

•  Accommodates  organizational  unique  criteria 

Allows  alignment  of  diverse  improvement  methods 

•  Focuses  all  efforts  to  support  organizational  objectives 

•  Guides  integration  and  prioritization  of  improvements 

Biggest  benefit  is  organizational  process  areas 

•  Infrastructure  basics  enable  re-organization  agility 

•  Again  -  links  process  management  &  improvement  to 
organizational  objectives 


Raytheon 

Customer  Success  Is  Our  Mission 


CMMI  10th  Annual 
Technology  Conference 

ft, 

Nancy  Fleischer 
Program  Manager 
Integrated  Product  Development  System 

16  November  2010 
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Raytheon 

Customer  Success  Is  Our  Mission 


CMMI  10th  Annual  Technology 
Conference- 

CMMI  Implementation  Strategies  for  Success 


Guiding  Principles: 

The  reason  to  implement  CMMI  is  to  improve  business  performance 
The  only  successful  CMMI  implementation  addresses  business  needs 
The  only  successful  measures  connect  to  business  performance  needs 
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Goals  of  CMMI  Implementation 


Raytheon 


■  Using  the  CMMI  should  improve  quality,  cost  and  schedule 
(productivity)  performance  of  your  organization 

■  A  few  published  improvements  in  Defense  Industries 

-  On-time  deliverable  increase  from  95%  to  99.9% 

-  6.35  times  less  defect  discovery  and  repair  hours  during  system  testing 

-  Schedule  Performance  Index  increase  from  .78  to  .93  over  three  years 

-  Cost  Performance  Index  increase  from  .88  to  .96  over  two  years 

■  See  “Benefits  of  CMMI  Within  the  Defense  Industry” 

-  Published  by  Software  Engineering  Institute,  Carnegie  Mellon  University,  Pittsburgh,  PA 
15213,  May  2010  ©  2010  Carnegie  Mellon  University 


-  http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense- 
Industry.cfm  V  V 


The  only  reason  to  implement  CMMI  is  to  improve  performance 
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Steps  in  Implementation 


Raytheon 


■  Determine  where  you  are  as  compared  to  the  model 

-  “If  you  do  not  know  where  you  are ,  a  map  won’t  help”-  Watts  Humphrey 

-  Most  companies  have  some  practices  that  are  consistent  with  the  model 

-  Find  the  right  behaviors  in  your  business  and  leverage  them 

■  Document  your  good  processes  that  are  in  place  but  are  not  recorded 

-  Use  pictures,  automated  tools,  text,  cartoons,  whatever  works 

■  Implement  processes  to  fill  gaps  that  address  business  needs  !!! 

-  CM  Ml  practices  are  not  processes 

-  Prioritized 

-  “Natural”  for  the  environment 

-  Integrated  into  how  you  do  business  targeted  your  value  stream 

-  You  are  ready  for-  maturity  means  experiencing  and  then  learning 

■  Monitor  penetration  and  institutionalization 

-  Growth  in  and  the  consistency  of  organizational  use  will  support  increasing 
organizational  capability 

The  only  successful  implementation  addresses  business  needs 
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CMMI  Implementation  Success 


Raytheon 


■  Effective  Use  of  CMMI  applies  processes  that  enhance  the  business 
based  upon  the  model* 

-  CMMI  is  a  model  for  process  improvement,  not  a  standard  and  not  a  process.  Adapt  to  business  environment  and 
available  resources  with  a  focus  on  performance  and  business  results 

-  Good  processes,  implemented  effectively  in  an  environment  of  continuous  process  improvement,  increase  the  likelihood 
of  achieving  success 

-  CMMI  can  help  resultant  processes  align  with  achieving  business  objectives  (Organizations  making  achievement  of 
CMMI  maturity  levels  their  primary  business  objective  may  not  achieve  benefits).  Focusing  on  areas  where  improvement 
is  needed  to  achieve  business  objectives  improves  acceptance. 

-  Advancements  in  CMMI  maturity  levels  should  be  based  on  a  business  case  for  improved  performance 

-  CMMI  maturity  level  ratings  are  useful  for  gauging  progress  in  achieving  organizational  process  improvement,  but 
ratings  are  not  alone  a  predictor  of  expected  project  performance.  If  a  project  or  organization  fails  to  consistently  utilize 
its  capability,  problems  will  occur. 

-  Do  not  use  CMMI  as  a  supplier  selection  tool  by  specifying  CMMI  maturity  levels 

-  Employ  appraisal  methods  of  various  types  and  levels  of  formality  to  identify  weaknesses  for  prioritizing  improvements 

*  Excerpted  from  “The  Economics  of  CMMI” 

http://www.sei.cmu.edu/library/abstracts/whitepapers/economics-of-cmmi.cfm 

■  Do  not  allow  outside  consultants  to  “change  the  way  you  do  business” 

■  Do  not  simply  copy  another’s  process,  it  might  not  work  for  you 


The  only  successful  implementation  address  business  needs 
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Measures  connected  to  Business  Needs 


Raytheon 


■  Using  CMMI  improves  the  business 


-  Use  typical  business  measures  over  the  long  term  to  determine  effectiveness 

Performance  (CPI,  SPI) 

Profitability  (Operating  Costs  versus  sales) 

Competiveness  (Win  Rate,  Sales) 

Timeliness  (Time  to  Market) 

Quality 


t 

1 

1  — — 1 

Other  measures  used  successfully:  dollarized  risk  of  non-c 
requirements,  first  pass  yield  predictions,  predicted  averag 
design  phase] 


lancet 
cost  [all  in 


-  These  big  measures  derive  “smaller”  measures 

•  Eg.,  productivity  and  EVMS  (Individual  work  package  budget  efficiency) 

•  Productivity  and  profitability  measures  create  focus  on  rework  and  defect  containment 
in  phase 

•  Leads  to  measuring  things  that  can  be  measured  in  a  smaller  time  spans 


-  Keep  the  connection  from  “small”  measures  to  “big”  measure  visible 


The  only  successful  measures  address  business  needs 
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Measures  connected  to  Business  Needs 


Raytheon 


■  Get  all  your  business  leaders  to  champion  these  measures 
and  the  connections 

-  Easier  said  then  done? 

■  Keep  measures  balanced 

-  Efficiency,  Quality,  Cycle  Time,  Timeliness 

-  Measure  all  aspects  so  that  one  aspect  does  not  override  the  others 

■  Increase  ROI  for  CMMI  from  improving  performance  (shown 
in  measures)  while  decreasing  the  cost  of  implementation 

■  Some  measures  we  have  used  effectively: 

-  CPI/SPI  -  Drawing  Release 

-  Requirements  Volatility  ~~  Design  Margin  Index 

-  Defect  Containment/Density  ~~  Size  and  Productivity 

-  Rework  (Drawing  re-release)  ~~  Staffing 

_  roI  -  On-time  Delivery 


The  only  successful  measures  address  business  needs 
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CMMI  10th  Annual  Technology  Conference- 

CMMI  Implementation  Strategies  for  Success 


Questions? 

Nancy  Fleischer 

nlfleischer(a)ra  ytheon.  com 

310-400-4490 


"The  most  important  business  indicators  are  our  performance  to  our  business  goals,  and 
in  most  cases,  the  activities  that  we  mapped  to  CMMI  are  the  same  activities  that  are 
strong  contributors  to  our  business  performance  and  whether  we  are  meeting  those 
business  goals." 
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ASA(ALT)  System  of  Systems 
Engineering  Processes 

CMMI  Technology  Conference 

16  Nov  2010 

Implementation  Strategy  &  the  CMMI® 


Mr.  Terry  Edwards 
Director,  ASA(ALT)  SoSE 
(703)  614-4540 
terry.edwards@us.army.mil 
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The  Army’s  View  of  CMMI... 

I 


WE'RE  GOING  TO  USE 
CMMI.  IT'S  A  MODEL 
FOR 

PROCESS 
A 


OR  IT  MIGHT  BE  ft 
PROCESS  FOR  CREATING 

TO 
ODEL. 


CMMI  has  arrived,  and  we  have  embraced  it !!! 
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Strategic  Environment 


ASA  (AL&T) 


•  Operational 

>  Persistent  conflict, 

>  Hybrid  threats  requiring  hybrid 
solutions 

>  Advanced/improvised 
technologies  targeted  against  the 
warfighter 


•  Budget 

>  Pressure  to  cut  defense  &  other 
spending 

>  Topline  base  budget  expected  to 
have  modest,  but  steady  growth 

>  “Do  more  without  more” 

>  Reduce  lifecycle-costs 


•  Army  Modernization 

>BCT-centric 
>Buy  fewer,  more  often 
incremental  fielding  of  capability 
thru  ARFORGEN 

—  SoSE 

Army-SOSE-CMMI  Conf  -111610 


•  Acquisition  Reform  DoD  5000 

> Increased  competition  throughout 
acquisition  process 

>Reduced  tolerance  for  cost/schedule  risk 
> Revised  Milestone  certification  reqs 

> Foster  innovation 
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UNCLASSIFIED  DOMINATE 


UNCLASSIFIED 


System  of  Systems  Engineering  (SoSE)  -  Environment 

ASA  (AL&T]  — 


Scope 


7SS/Q/7  The  Mission  ofASA(ALT)  SOSE: 


Tactical 

Domain 


Business 

Domain 


Infrastructure 
Domain 


Build  the  Bench 


Organize  the  Army 
Engineering  Community 
Establish  Technical  Authority 
and  engineering 
expertise/capability 
Pursue  accreditation  and 
certifications  of  organic 
workforce  &  organizations 


Create  Data  Transparency 

Establish  CM  and  an 
authoritative  repository  for 
products 

Establish  a  collaboration 
environment 

Establish  a  common  operating 
environment  for  engineering 
Products 


Provide  the  Army’s  leadership  and  materiel 
developers  with  the  necessary 
engineering/architectural  products  to  manage 
and  shape  the  Army’s  materiel  portfolio,  to 
ensure  a  System  Engineering  discipline  across 
the  Materiel  developer  community  throughout 
the  acquisition  life  cycle  and  grow  the  System 
Engineering  capability  within  the  Army  - 
through  education,  engineering  policy, 
guidelines  and  adoption  of  best  industry 
practices—. ’’Build  the  Bench  _ _ 

a 

Manage 


k# 


<3 


the  Army’s 
Portfolio 


% 


\2 

Enable  the  Process 


Establish  the  engineering  process  to 
deliver  synchronized  capability 
Establish  the  analytical  structure  with 
models  &  simulations 
Establish  the  engineering  compliance 
structure/process  for  acquisition 
execution  excellence 
Deliver  engineering  support  to  HQ 
staff  and  acquisition  community 


2011/12  13/14  15/16 


Manage  the  Portfolio 

Support  the  prioritization  of 
capabilities  within  the  portfolio 
Set  the  baseline  architecture 
roadmap  over  time 
Support  the  resourcing  process 
Synchronize  and  align  the  S&T, 
systems  integration,  test,  and 
certification  activities 


Ver.  2.7 


^eedbac/c 
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Organizing  the  SoS  Space 


A 


ASAjAL&T) 


AAE' 


MilDEP 


EXECUTIVE 

COUNCIL 


DAS-M  Chair 


ENGINEERING 
OVERSIGHT 

SoS£C/Ia/rC0UNCIL(E°C)  DPEOs 


Configuration 

Steering 

Board 


\-PEO 

-SoSE 

-S&T 

-  INTEGRATION 
|  - RDECOM 
- FFRDC 


AVIATION 


MISSION 

COMMAND 


FIRES/AMMO 


■■MB, 


TRANSPORT 
&  NETOPS 


Tactical  Vehicle/ 


Sustainment 


+m 


NETWORK 

INTEGRATION 


MISSION  COMMAND 
INTEGRATION 


Ground  Combat 
Systems 


SoSE  Cross-IPT 
Integration  WGs 


INTEG  BASE 
DEFENSE 


INTEL  & 
SENSORS 


PLATFORM 

INTEGRATION 


N 

DOMAIN  SE  IPTs 


JOINT  COMBAT 
ID 


PEO  TIER  2 
SE  GOVERNANCE 


Business 


Chem  /  BIO 


SIM  & 
TRAINING 


Concept 


ft 


Authoritative 
Data  Sources  o 


Authoritative  System 
Architectures 


S¥ST1=M5 

«tCHn-|&CTTJFt£ 


Acquisition 

Analysis 


V  System  A  |  f"  HmSm  |-  System  A  1 

_ /  ^System  B  |  ^  Capability  j  ^System  B 


<^= 
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The  ‘Network’ 


GLOBAL  INFORMATION  GRID 


INFOJMATK 


Typical  ‘Network’  Layers 

•  Sensors 

•  Applications 

•  Services 

•  Transport 

•  Standards 
Added 


Applications 
&  Services 


•  Force  Structure 

•  NETOPS 


LANDWARNET 

_ 


•  Sutt«  of  jioi'Dir,  nanedunmaned  platforms, 
with  a  drverse  set  of  sensors  taitaed  tome 


FHCEKl 


■  Battle  Command  and  Control.  Inteltowce. 
Surveiance.  and  Reconnaissance  (158). 
Embedded  Train  ng.  and  Sustument 


APPLICATIONS 


•Common  toolset  of  irfrastructw  prices. 


Z 

0 

h 

< 

£  'Multi-Tiered  Ground.  Air.  Space 
u-  Dynamic.  On  the  fdove  CommmicMoiK 
0  NeTwoik 

z  _ _ L 

• '  "cztrcT  ’  tKm 

’EEPOFT 

0i91 


'  i  ■  ■  Ilf: 


TRANSPORT 

Teiecommuni :  stions) 


•  Common  Warfighter  Interface  Display  to  FCS 

•  Presents  relevant  and  fatly  battlefield  Mo 

•  Provides  content-senslhe  presentation  irvjnf 
•Reduces  training  demands 


•Platfoimhtegration 
;  •  TactkM  Me>A  or+  Integration 
•SbitegkNetAorkGIG  Integrate 
Badtte  Commarvj  Integration 


1  -PiovKlesseaiie.  reliable  access 
•  Infiasbuctue  embedded  n  H« 

,  mob*?  platforms  and  moves 
•  wltfittv?  combat  formations 
•Unprecedented  range,  capacity 
;  and  dependability  ei 
communac  aliens 


RffETS 

COMERCWLFBER 

Foundation  of  the  FCS  (BCT)  network 
Provides  the  governance  for  which  the 
other  layers  are  shaped  aivi  formed 


l.ClrJfc 


MUfTtCJRt 

BIGIEEtl 


STANDARDS 


POUC 


“The  Network  is  the  singularly  most  important  program  to  the  Army” 

—  GEN  George  W.  Casey,  Jr.  23  July  2010 

•n  -_7"_  T  UV1  ▼  fe.T% 


i  ocac  ' — 

Army-SOSE-CMMI  Conf  -111610 


UNCLASSIFIED 


DOMINATE 


6 


NETWORK  ORE  RATIONS 


UNCLASSIFIED 


Army  Network  -  Enterprise  View 


ASA  (AL&T) 


Satellite 

Transport 


A  Home/TJ 


^  LandWarNet 

Core 

Area  Processing  Center  NOSC 


v  Internet 


Teleport/STEP 
\  FRHN 1/ 


Post/Camp/Station 


Tactical 


Garrison 


-V  -%  j 

-<i“  JTF/^ 

(Corps/^SB) 


Transport 

Tactical 


Reset  (Phases  0-1) 


Transition 


Deploy  (Phases  2-5) 


S<* 
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COE  Architecture  Guidance 


DEPARTMENT  Of  THE  ARMY 

WASHIhQTCw  be  jojio 


KT  20  Ml 


MEMORANDUM  FOR  SEC  DISTRIBUTION 

SUBJECT:  Common  Qpe-/a1irtg  Environment  A/chlbadurifr  Guidance 
".  Re'erences: 


a.  Memorandum.  Vies  Chief  of  SleH  of  Bte  Anny  (YCSA}.  subject:  Ashling  Army 
IMnhwprk  and  Battle  Commaivd  Modernization  Objectives,  dated  26  December  2CJ[]g 

b  Documanlr  CIQ/G-6,  1i<led-  Common  Operating  Envjro*wiien1  Arotirtedura. 
(Appendix  0  to  Guidance  for  'End  Slate'  An»v  Enterprise  Netwoifc  Architecture)  riatort  1 
October  20(0 


2.  The  ClO/G-G,  in  dose  coordinatfon  with  A3A(ALT)  Syatema  of  Systems  Engineering 
(SOGEJi,  has  developed  Ihe  Common  Operating  Environment  iCOE)  Aichiteotum 
concepl. framework.  and  standards  for  Ihp-  Army  Enterprise  Network.  in  addition,  |n 
order  to  inform  Program  Objective  Memorandum  13^17  investment  decisions,  ASA(ALT|i 
wifT  publish  a  com  pieman  I  ary  Implementation  Plan  that  deecrbos  the  step? 
schedule  Tor  bringing  Army  systems  into  oempfiancar  whh  the  COE  Architecture 
guidance  HeneelorTh.  compliance  with  (he  COE  Amhitecture  and  Irciplementaliiwi  Plan, 
will  be  mandatory  ter  ai  programs  undertime  purview  pi  Hhe  Army  Acquisition  Executive. 

3.  The  ClOriSi-S  and  ASA^AO)  ana  committed  bo  enabling  Ihe  An-qy  to  produce  high- 
quality  *pplicafons  rapidly  write  reducing  (he  complexities  embedded  m  (ha  design, 
development,  lasting  end  depknymanl  cycle.  The  CO£  Architecture  and  6  m  pie  men  tali  on 
Plan  will  provide  direction  to  our  indu^ry  panners  regarding  our  framework,  standards. 
Bolh  documents  ere  GWWklerad  Id  be  living  inelm^nenls  and  Will  continue  te  evdtve  in  a 
ooordinaled  manner  in  ordar  to  keep  up  with  Ihe  rapid  changes  in  (achnology- 

*■  Our  poinis  o-t  cofilacttor  this  memorandum  are:  COL  Anthony  Howard.  Sr.,  <7031 
6CM-20EB  or  enlhony.hewardsr6us.army.mii:  end  Mr.  Phillip  Miner,  £7G3fr  604-7133  or 
ph^j,  minor®  UB.airmy.mil. 


FFREY  A.  SORENSON 
doutenanl  General,  OS 
■Chiet  Iniorm^tion  Oflioer/G^6 


TYlaJtoim  R.  O'Nedi  ' 


Assiatanl  Secretary  of  the  Army 
(Acquisition.  Logistics  and  Technology) 


•  CIO/G-6  in  close  coordination  with 
ASA(ALT)  Systems  of  Systems  Engineering 
(SOSE)  has  developed  COE  Architecture 
Guidance 

•  COE  Architecture  Guidance: 

-  Defines  the  COE  and  Computing 
Environments 

-  Describes  the  CEs  architecture  and  services 

-  Specifies  COE  principles  and  technical 
architecture  standards 

-  Details  a  maturity  model  for  cost-benefit 
analysis  trades  and  to  evaluate  programs’ 
alignments  with  COE 

•  ASA(ALT)  will  develop  COE  Implementation 
Plan: 

-  Inform  Program  Objective  Memorandum 
(POM)  13-17  investment  decisions 

-  Identify  the  implementation  strategy,  time 
lines,  effective  dates  and  key  milestones  for 

mn\/inn  Arm\/  r\/otAnno  trv  th/h  PfiC 


“Establishing  'left  and  right  limits' . . . 

Chiarelli  Touts  Common  Operating  Environment  Architecture  At  AUSA 


-  By  Tony  Bertuca,  Inside  the  Army,  October  29,  2010 

SoSE 


Army-SOSE-CMMI  Conf  -111610 


UNCLASSIFIED 


DESIGN  *  DEVELOP  ”  UEUVHt  w 
DOMINATE 


8 


UNCLASSIFIED 


Army  needs  a  Software  “Eco-System 

ASA  (AL&T) 


jj 


Software  Ecosystem  defined  as  a  set  of  businesses  functioning 
as  a  unit  and  interacting  with  a  shared  market 
for  software  and  services,  together  with  relationships  among 
them.  These  relationships  are  frequently  underpinned  by  a 
common  technological  platform  and  operate  through  the 
exchange  of  information,  resources,  and  artifacts  David  G.  Messerschmitt  and 

Clemens  Szyperski  (2003).  Software  Ecosystem:  Understanding  an  Indispensable  Technology  and  Industry.  Cambridge,  MA,  USA:  MIT 
Press. 


An  Army  Eco-System  would  need  to  provide: 

•  Improved  agility 

•  Reduced  life  cycle  costs 

•  Adaptability 

•  Means  to  address  cyber  threats 
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%)  Realizing  the  Army  Software  Eco-System 

-—ASA  (AL&T) 

Eco-System  Realization:  A  Common  Operating  Environment 
(COE) 

Common  Operating  Environment: 

Automation  services  that  support  the  development  of  the 
common  reusable  software  modules  that  enable 
interoperability  across  multiple  combat  support  applications. 

This  includes  segmentation  of  common  software  modules  from 
existing  applications,  integration  of  commercial  products, 
development  of  a  common  architecture,  and  development  of 
common  tools  for  application  developers. 

Dictionary  of  Military  and  Associated  Terms.  US  Department  of  Defense  2005. 
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But,  an  Army  COE  Must 


Operate  across  families  of  computing  environments  (CE): 
•  Data  Center  /  Cloud 


Command  Post  Enterprise 

Real-Time,  Safety-Critical,  &  Embedded  server 


Tactical 

server 


•  Mounted 

•  Mobile/Handheld 

•  Sensors 

AND 


Platform 


ra 


Mobile 


Common  Operating 
Environment 


Client 


4 


Sensors 


•  Improve  agility:  In  development,  acquisition  &  operations 

•  Reduce  life  cycle  cost:  In  both  new  and  legacy  applications 

•  Be  adaptable:  To  changing  standards  across  all  Army  systems 

•  Address  cyber  needs:  Keep  pace  with  ever  changing  threats 
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Computing  Environment  Example 

ASA  (AL&T)  (Mobile  Handheld) 


Data  Center-CE/Com™3"^  post-CF 


LU 

o 


T3 

0 


"D 

C 

CD 

X 


0 


Q 

O 


User-Specified  Apps  Could  be  PEO/PM  purchased/developed 
Could  be  User  configured/programmed  Could  be  unique  unit  service 
LandWarNet  2010  Apps4Army 


Standard  Apps 

Internet,  Native,  Hybrid,  Web 


PDK=Standard  set  of 
core  assets, 

APIs,  libraries, 
product  builders, 
documentation 


Core  Services  / Data 


Data  Caching  SVCs 


Transmission  SVCs 


UmreSVCs 


]  I  Man  SVCs  !□ 


Web  Tech  SVCs 


VMF  SVCs 


Operating 

System 


*  iPhone 


CORE  Physical 


Ruaaedized 


Wearable 


COTS 

Capability 


Peripherals 


SECURITY 


<> 

TT 


IT 


8, 


X 

CD 

"D 

C 

CO 


.o 

lo 

CD 


User-Loaon 

9 

1 

Data  at  Rest 

r 

Encrvotion 

r- 

river 
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CE  Relationships  and  Boundaries 


■  Data  Center-CE  /  Command  Post-CE 


Command  Post  -  CE 

Data 


DATA  Center-CE 

Enterprise  App  Store 

Services 

(Map,  BlueForce,  VMF) 

Pavload  upload 


Gateway 

Payload 

upload 


DATA 
Gateway 


SOAP,  REST, 
KML,  UCORE, 
VMF  RSS,  SMS 


Platform-CE 


Mounting,  Power 


Data  Gateway 


payload  upload 


data 

Gateway 


Wearable 


Ruggedized 


COTS 


Android/iPhone 


D 


Mobile  Hand  Held  CE 


Function  Objectives:  Connect, 
Communicate,  Collect, 

Share,  collaborate,  local  Situation 
Awareness 


rss  ^ap  lma9ery  uto 


Sensor-CE 


USB 

Serial 


Heart  beat  monitor 

Biometric  capture 


Ethernet 

WiFi 

Blue  Tooth 
ZiGBE . 


COTs 


■U 


Peripherals 
Keypad,  smart  pen, 
visualization 
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Adaptability  Game  Changer 


ASA  (AL&T) 


y  > 

Data  at  Rest,  User 

SMS,  e-mail, 

COIN,  SUGV 

ID,  Encryption 

Safari,  WebKit 

Controller 

over  the  Air 

AES  256 

Others: 

Cascade  Style  Suite  (CSS)  3.0 
Super  Java  Script 
Unified  Font 
HTML  5.0  Secure  Web 
Operating  Systems 
Android  2.2  (Java,  C,  C++) 
iOS  4.2  (objective  C) 

Research  in  Motion  5.0  (C?) 


User/Device 
Management 

Exchange  Active  Sync 


Adaptability  Game  Changer 
Attributes 

•  Provides  ability  to  access  enterprise 
application  store 

•  Provides  ability  to  rapidly  deliver 
mission  specific/soldier  centric 
capabilities 

•  Provides  the  ability  to  let  the  soldier 
tailor  different  applications,  widget 

to  meet  their  function,  task,  condition, 
standard  for  mission  success 

•  Enables  short  release  cycles  of 

functional  capability  (deployed  as 
apps) 

•  Enables  flexible  delivery  of 
capability 

•  Enables  user  or  3rd  party 
contributions  of  capability  (through 
the  Enterprise  App  Store) 
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fp  Will  a  COE  work? 

i - '  ASA  (AL&T) 

It  has  to,  because  today... 

•  Software  has  value  only  in  the  context  of  the  system  it  was  developed  to 
support  (and  the  contractor  who  developed  it) 

•  Software  Integration  &  Interoperability  have  become  intractable 

•  It  takes  too  long  to  capitalize  on  commercially  available 
solutions/innovations 

It  can  if... 

•  The  implementation  starts  with  a  minimum  set  of  standards 

•  Standards  &  common  services  are  planned  to  evolve  continuously  (with 
appropriate  resources)  aligned  with  Army  goals  &  objectives 

•  Compliance  is  incentivized  (and  enforced) 

•  The  processes  for  managing  the  COE  are  disciplined,  transparent  and 
support  application  developers 
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How  CMMI  Can  Help 

ASA  (AL&T) 

CMMI  provides  a  model  that  can  support  the  disciplined 
enterprise-level  implementation  of  the  processes  needs  to 
develop,  evolve,  and  deliver  the  common  services  required  by 

the  COE:  Used  for 

managing 
interfaces/solicitat 
ion  of  needs  from 

Used  to  application 

implement/evolve  developers 

common  services 


Used  to  buy 
commercial/indus 
try  supplied 


services 
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CMMI 

—  ASA  (AL&T)  — 


The  Value  Proposition  (i) 


Goal  -  Reduce  Armament  SEC  software  life 
cycle  costs  by  identifying  and  fixing 
defects  closer  to  phase  of  origin 
(requirements,  design,  code, 
integration,  test) 

o  Established  Armament  SEC  baselines  and 
models  using  industry-standard  Defect 
Containment  Matrix  (DCM)  methodology 

-  Tracked  defect  “phase  leakage”  (finding 
defects  in  phases  after  the  originating  phase) 


Life  Cycle  Phase  Discovered 


L 

R 

D 

C 

1 

T 

Legacy 

10% 

29% 

7% 

4% 

29% 

17% 

14% 

TJ 

4> 

Requirements 

60% 

75% 

5% 

16% 

2% 

2% 

W 

C 

Design 

4% 

67% 

16% 

0% 

17% 

» 

Code  and  Unit  Test 

22% 

89% 

3% 

8% 

0 

Integration  Test 

2% 

66% 

34% 

Test 

2% 

100% 

100% 

3% 

|  46%  | 

6% 

|  32% 

|  5% 

8% 

In  the  red 

64% 

20% 

17% 

8% 


o  Maturity  Level  5  projects’  focus: 

-  Optimize  within-phase  verification 
processes  (e.g.,  Peer  Review, 
Unit  Testing,  etc) 

-  Leverage  reuse  of  mature  code 

-  Increase  automation  of  testing 


Defects  Phase  Containment  /  Leakage 
(High  Severity  Defects  -  Priority  1,  2  &  3) 


Captured  within  Leakage  =  1  Leakage  >  =2 

Phase 

Cost  avoidance  realized: 

■  Less  rework  late  in  life  cycle  when  it  is  most  expensive  to  repair 

■  Resulting  in  reduced  schedule  risk 
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CMMI 

—  ASA  (AL&T)  — 


The  Value  Proposition  (2) 


Defect  Density  by  Priority  (PCR) 
(High  Severity  Defects  Only) 
(Ave  Project  -  219  KSLOC) 

0.800 


Average  Hours  per  Defect  per  Phase 
to  Repair  (High  Severity  Defects) 


Priority  1  Priority  2  Priority  3 


Highest  severity  defects  leaked  are  decreased  by 
at  least  79%: 

■  More  rigorous  peer  reviews  focusing  on  systemic  issues 

■  Broadened  participation  and  tailored  role-based  review 
criteria 


120.0 

100.0 

80.0 

60.0 

40.0 

20.0 

0.0 


24%  Fewer 


Requirement  &  Code  &  Unit  Test  Integration  & 
Design  Formal  Test 


Defects  discovered  in  ML5  projects  cost  less  to 
repair  in  all  phases  on  average  -  in  particular, 
Integration  &  Test  phases  achieved  a  24% 
reduction  of  hours  expended 
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Build  the  Credentials  of  the  Organization 

ASA  (AL&T) 


Pursue  improvement  opportunities  across 
the  Army  acquisition/engineering 
community 

Share  best  practices  among  diverse  Army 
acquisition/engineering  organizations 
Leverage  high  performing 
acquisition/engineering  organizations 
Army  Systems  Engineering  Forum 
Army  Strategic  Software  Improvement 
Program 


i  i  >  *-[#>•  V  i  S  *. 

C*Kt*<IMC  :4*<«  pH  <r<\  CU/1 


CDF  Rayleigh  Curve 


Percent  50% 

40% 

30% 

20% 

10% 


TARDEC  SEC 
Warren,  Ml 


ARDEC  SEC 
Picatinny,  NJ 


w _ 

*1  CECOM  SEC/CERDECSED  Ft 
w  Monmouth,  NJ/Aberdeen,  MD 


Huntsville,  AL 
AMRDEC  SED 
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BUT  -  This  is  Uncharted  Territory 


ASA  (AL&T) 


•  There  is  scant  data  about  actual  use  of  the  CMMI 
constellations  in  common  operating  environments, 
which  suggests 

-  It  hasn’t  been  done  before 


-  If  it  has  been  done,  the  results  are  being  held  proprietary 


We’d  love  to  hear  your 
thoughts  & 
experiences... 
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CMMI  Transition  Status 
Reported  to  the  SEI  as  of  10-31-10 


Training 

Introduction  to  CMMI  VI  .2 

120,838 

Intermediate  Concepts  of  CMMI 

3,238 

Understanding  CMMI  High  Maturity  Practices 

636 

Introduction  to  CMMI  VI  .2  Supplement  for  ACQ 

1,325 

Introduction  to  CMMI  VI  .2  Supplement  for  SVC 

2,361 

Introduction  to  CMMI  for  Services  VI  .2 

314 

Certifications 

Introduction  to  CMMI  VI  .2  Instructors 

408 

CMMI-ACQ  VI. 2  Supplement  Instructors 

66 

CMMI-SVC  VI. 2  Supplement  Instructors 

131 

Introduction  to  CMMI  for  Services  VI  .2  Instructors 

23 

SCAMPI  VI  .2  Lead  Appraisers 

466 

SCAMPI  VI. 2  High  Maturity  Lead  Appraisers 

142 

CMMI-ACQ  VI  .2  Lead  Appraisers 

72 

CMMI-SVC  VI  .2  Lead  Appraisers 

147 
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CMMI  Adoption 


"  T  W\ 

USA 


Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

425 

35.2% 

3354 

91.8% 

651 

54.0% 

223 

6.1% 

130 

10.8% 

76 

2.1% 

1206 

100.0% 

3653 

100.0% 

>200, 

22.4% 

101-200, 

17.3% 


<100, 
60.2%  < 


>2000,1.6% 
1001-2000,3.3% 
501-1000,5.0% 
301-500,  6.1% 
201-300,  7.3% 

101-200,  17.3% 
76-100,  8.3% 
51-75,  12.6% 

26-50,  22.0% 

<25, 17.3% 


Organization  Size  (Employees) 
(4863  organizations  reporting) 


Services  (72.1%) 

•  Business  Services 

•  Engineering  and 
Management  Services 

•  Health  Services 

•  Other  Services 


Manufacturing  (15.7%) 

Electronic  and  Electric  Equipt 
Transportation  Equipment 
Instruments  &  Related  Products 
Industrial  Machinery 
Other  Mfg  Industries 

Other  (12.2%) 

•  Finance,  Insurance,  Real  Estate 

•  Public  Administration/Defense 

•  Transportation,  Communication, 
Utilities 


Commercial 

In-House 

Contractor 
for  Military/ 
Government 

Military/ 

Government 

Agency 

No  Rating  Given 

5.3% 

8.0% 

22.3% 

Initial  (ML1) 

0.6% 

1.4% 

1.0% 

Managed  (ML2) 

25.8% 

31.5% 

45.6% 

Defined  (ML3) 

58.1% 

49.3% 

26.7% 

Quantitatively  Managed  (ML4) 

2.9% 

1.0% 

1.5% 

Optimizing  (ML5) 

7.2% 

8.9% 

2.9% 

(3779  orgs) 


(874  orgs) 


(206  orgs) 


Source:  SEI  Process  Maturity  Profile,  Sept  2010. 

httn 7/www  ^pi  pmu  pdu/pmmi/pp^pcitijrlipci/nrnfilpci/pmmi  pfm 
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CMMI  VI. 2  Foreign  Language  Translation  Status 
Reported  to  the  SEI  as  of  10-31-2010 


Language 

Status  (for  CMMI-DEV  VI  .2) 

Japanese 

Chinese  (trad.) 

French 

German 

Spanish 

Portuguese 

Completed  August  2007.  Intro  course  translated  October  2007 

Completed  December  2007 

Completed  August  2008 

Completed  April  2009.  Intro  course  translated  October  2009 

Completed  in  June  2009 

Completed  in  May  2010 

Language 

Status  (for  CMMI-ACQ  VI  .2) 

Chinese  (trad.) 

Completed  April  2009 

Language 

Status  (for  CMMI-SVC  VI  .2) 

Chinese  (trad.) 

Arabic 

Completed  in  July  2010 

To  start,  pending  agreement 

Language 

Status  (for  CMMI-DEV  VI  .3) 

Dutch 

Underway 
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Number  of  SCAMPI  vX  Class  A  Appraisals  Conducted  by  Year 
by  Representation*  Reported  as  of  10-31-10 

*Where  Representation  is  reported- 


1600 


1400 


1200 


1000 


Staged 

Continuous 


0 


2004 


2005 


2006 


2007 


2008 


2009  10/31/2010 
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CMMI  Model  Structure 


Incremental  Frameworks  for 
Continuous  Process  Improvement 


Benchmark  Ratings 


•Goals 

•Process  Areas 


•Maturity  Levels 
•Capability  Levels 


CMMI-DEV 


•Requirements  Development 
•Supplier  Agreement  Mgmt 
•Technical  Solution 
•Product  Integration 
•Verification 
•Validation 


CMMI-SVC 

•Capacity  &  Availability 
Management 
•Incident  Resolution  and 
Prevention 

•Supplier  Agreement  Mgmt 
•Service  Continuity 
•Service  Delivery 
•Service  System  Development 
•Service  System  Transition 
•Strategic  Service  Mgmt 


CMMI-ACQ 

•Agreement  Management 
•Acquisition  Requirements 
Development 
•Acquisition  Technical  Mgt 
•Acquisition  Validation 
•Acquisition  Verification 
•Solicitation  and  Supplier 
Agreement  Development 


CMMI  Model  Foundation  (Core  Process  Areas) 


•Requirements  Management 
•Project  Planning 
•Project  Monitoring  &  Control 
•Measurement  &  Analysis 
•Configuration  Management 
•Process  and  Product  QA 


•Policies 

•Plans 

•Resources 


•Integrated  Project  Management 
•Risk  Management 
•Decision  Analysis  &  Resolution 
•Organizational  Process  Focus 
•Organizational  Process  Definition 
•Organizational  Training 


•Quantitative  Project  Mgmt 


•Causal  Analysis  &  Resolution 
•Org  Process  Performance 
•Org  Performance  Management 


Institutionalization 


•Responsibilities 

•Training 

•Managing  Configurations 


•Stakeholder  Involvement 
•Monitoring  and  Control 
•Objective  Evaluation 


•Management  Visibility 
•Defined  Process 
•Improvement  Information 
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CMMI  Product  Suite,  Version  1.3 


Version  1 .3  focused  on  but  was  not  limited  to  the  following: 

•  High  Maturity 

•  Appraisal  efficiency 

•  Consistency  across  constellations 

•  Simplify  the  generic  practices 

Version  1 .3  was  change  request  (CR)  driven. 
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Comparison  of  Models 


Measure 

CMMI  for  Development 

CMMI  for 
Acquisition 

CMMI  for  Services 

VI. 1 
Staged 

VI. 1 
Cont 

VI  .2 

VI. 3 

VI  .2 

VI. 3 

VI  .2 

VI. 3 

Pages 

715 

710 

560 

468 

428 

423 

531 

506 

Process 

Areas 

25 

25 

22 

22 

22 

22 

24 

24 

Generic 

Goals 

2 

5 

5 

3 

5 

3 

5 

3 

Generic 

Practices 

12 

17 

17 

13 

17 

13 

17 

13 

Specific 

Goals 

55 

55 

50 

49 

46 

47 

52 

53 

Specific 

Practices 

185 

189 

173 

167 

161 

163 

182 

181 
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Achieving  Maturity  Levels 


Visibility  into  the  Team’s  Performance 


Operational 
Need 


Acquirer 


CMMI  for  Acquisition 
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Capability  Profiles  with  Multiple  Constellations 


7 — 71 - 

7 - 71 - 

7  71 - 

7 - 7 - 

7  7 - 

_ ^  1 _ 

1 _ z _ 

i 


z 


7 


VER  VAL  AVER  ATM  CAM  PPQA 


CMMI VI  .3  and  Beyond 

Software  Engineering  Institute  Carnegie  Mellon  pmiMps- juiy  2010 

'  ©2010  Carnegie  Mellon  University 


CMMI  Appraisal  Methods 


Software  Engineering  Institute 


Carnegie  Mellon 
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Common  Themes  to  SCAMPI  Improvements 


Scoping  Appraisals 

•  Confusion  caused  by  “focus-”  and 
“non-focus”  projects 

•  Minimum  scoping  rules  for  a  wide 
range  of  organization  types 

Collecting  Data 

•  Confusion  caused  by  “direct”  and 
“indirect”  artifacts 

•  Data  sufficiency 


Pain  Points  that  Make  SCAMPI  Difficult  to 
Sustain 

•  Need  to  achieve  efficiency 

•  Expanding  organizational  scope 

•  True  cost  of  PI  IDs 

Attaining/Maintaining  Appraisal  Ratings 

•  Period  of  validity 

•  Multi-constellation  appraisals 
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Appraisal  Transition 


Once  CMMI  Version  1.3  is  released: 

•  During  a  period  of  one  year,  organizations  may  use  either  VI  .2  or 
VI  .3  models  for  their  appraisals  until  November  30,  201 1 ,  except  for 
uses  dependent  on  translated  versions  of  products  that  are  approved 
in  advance. 

•  SCAMPI  A  MDD  VI  .2  may  be  used  until  November  30,  201 1 . 

•  All  appraisals  will  be  valid  for  3  years. 
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CMMI  Training 


Software  Engineering  Institute 


Carnegie  Mellon 
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Training  Updates 


The  following  courses  will  be  updated  for  Version  1 .3: 

•  Introduction  to  CM  Ml -DEV 

•  Introduction  to  CMMI-SVC 

•  ACQ  Supplement  for  CMMI-DEV 

•  SVC  Supplement  for  CMMI-DEV 

•  SCAMPI  Team  Training  &  SLAT 

•  Advanced  DEV  courses 

New  courses: 

•  Development  Supplement  for  CMMI-SVC 

•  Advanced  courses  for  CMMI-SVC 
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Training  Transition 


The  SEI  will  provide  on-line  upgrade  training: 

•  Users  make  the  transition  by  taking  the  upgrade  course. 

•  Instructors  make  the  transition  by  taking  upgrade  course. 

•  Lead  Appraisers  make  the  transition  by  taking  upgrade  course  and 
specific  SCAMPI  method  upgrade  training. 
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VI .3  CMMI  Model  Updates: 

Core  PAs 


Software  Engineering  Institute 


Carnegie  Mellon 
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VI  .3  Changes  to  GGs,  GPs,  and  GP  Elaborations 


Positioned  generic  goals,  generic  practices,  and  GP  elaborations  in  one 
central  location  as  the  first  section  of  Part  2  in  all  three  models. 

Simplified  GG1  to  make  it  more  readable. 

Renamed  GP  2.6  to  “Control  Work  Products.” 

Added  “selected  work  products”  to  the  GP  2.9  statement. 

Simplified  the  GP  3.2  statement  to  replace  “collect  work  products, 
measures,  measurement  results,  and  improvement  information”  with 
“collect  process  related  experiences.” 

Eliminated  GG4  and  GG5. 
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Core  PAs:  Support  Category 


Configuration  Management 

Establish  and  maintain  the  integrity  of  work 
products  using  configuration  identification, 
configuration  control,  configuration  status 
accounting,  and  configuration  audits 

Decision  Analysis  and  Resolution 

Analyze  possible  decisions  using  a  formal 
evaluation  process  that  evaluates  identified 
alternatives  against  established  criteria 

Measurement  and  Analysis 

Develop  and  sustain  a  measurement  capability 
used  to  support  management  information  needs 

Process  and  Product  Quality  Assurance 

Provide  staff  and  management  with  objective 
insight  into  processes  and  associated  work 
products 


CM:  Clarified  that  CM  can  apply  to 
hardware,  equipment,  and  other 

tangible  assets. 


DAR:  Added  guidance  on  defining 
the  scope  of  the  decision  and 
communicating  results. 


MA:  More  clearly  distinguished 
between  information  needs  and 
objectives,  measurement 
objectives,  and  business/project 
objectives.  Included  a  table  of 
examples  (as  in  ACQ)  for  DEV 
and  SVC. 

Clarified  that  PPQA  also  applies  to 
organization  level  activities  and 
work  products. 
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Core  PAs:  Process  Management  Category 


Organizational  Process  Definition 

Establish  and  maintain  a  usable  set  of 
organizational  process  assets,  work  environment 
standards,  and  rules  and  guidelines  for  teams 

Organizational  Process  Focus 

Plan,  implement,  and  deploy  organizational  process 
improvements  based  on  a  thorough  understanding 
of  current  strengths  and  weaknesses  of  the 
organization’s  processes  and  process  assets 

Organizational  Training 

Develop  skills  and  knowledge  of  people  so  they  can 
perform  their  roles  effectively  and  efficiently 


Converted  goal  on  teaming  to  a 
single  practice,  which  is  no  longer  an 
“addition”  for  IPPD  only. 


Simplified  SP  3.4  to  replace  “process- 
related  work  products,  measures,  and 
improvement  information”  with  “process 
related  experiences”. 


Expanded  applicability  to  training 
development  and  delivery  methods 
such  as  self  study,  mentoring,  and 
online  training. 
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Core  PAs:  Project  and  Work  Management 
Category  -1 


Integrated  Project  Management 

Establish  and  manage  the  project  and  the 
involvement  of  relevant  stakeholders  according  to 
an  integrated  and  defined  process  that  is  tailored 
from  the  organization’s  set  of  standard  processes 


Simplified  SP  1 .7  to  replace  “work 
products,  measures,  and  documented 
experiences”  with  “process  related 
experiences.” 

Converted  goal  on  IPPD  or  Integrated 
Teaming  to  a  single  practice  (IPPD  no 
longer  an  addition). 


Project  Monitoring  and  Control 

Provide  an  understanding  of  the  project’s  progress 
so  that  appropriate  corrective  actions  can  be  taken 
when  the  project’s  performance  deviates 
significantly  from  the  plan 


Added  guidance  for  monitoring 
risks,  data  management, 
stakeholder  involvement,  project 
progress,  and  milestone  reviews. 


Project  Planning 

Establish  and  maintain  plans  that  define  project 
activities 


Added  guidance  on  determining 
project  lifecycle  and  milestones. 

Added  subpractices  on  determining 
data  rights  and  need  for 
configuration  control,  and 
determining  communication 
requirements  and  other  continuing 
resource  needs. 
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Core  PAs:  Project  and  Work  Management 
Category  -2 


Requirements  Management 

Manage  requirements  of  the  project’s  products  and 
product  components  and  to  ensure  alignment 
between  those  requirements  and  the  project’s  plans 
and  work  products 


Risk  Management 

Identify  potential  problems  before  they  occur  so  that 
risk  handling  activities  can  be  planned  and  invoked 
as  needed  across  the  life  of  the  product  or  project  to 
mitigate  adverse  impacts  on  achieving  objectives 


Changed  the  focus  of  SP  1 .5  so  that 
it  now  reads  “Ensure  that  project 
plans  and  work  products  remain 
aligned  with  requirements.” 


Included  examples  related  to: 
architectural  risks,  use  of  industry 
standards  to  identify  risks,  FMEA, 
and  consequence  monetization. 

Provided  guidance  on  maintaining 
risk  parameters  through  life  of  the 
project. 
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SAM  -  the  Shared  PA 


SGI:  Establish  Supplier  Agreements 

SP  1 .1  Determine  Acquisition  Type 
SP1.2  Select  Suppliers 

SP  1 .3  Establish  Supplier  Agreements 
SG  2:  Satisfy  Supplier  Agreements 

SP  2.1  Execute  the  Supplier  Agreement 

SP  2.2  Accept  the  Acquired  Product 

SP  2.3  Ensure  Transition  of  Products 


Clarified  the  applicability  of  SAM 
practices. 


Demoted  SP  2.2  and  SP  2.3  to 
subpractices  of  SP  2.1  and 
renumbered  the  remainder  of  the 
practices. 


Revised  SP  2.3  to  allow  its 
applicability  to  times  when  the 
product  or  service  is  delivered 
directly  to  the  customer  or  end 
user  from  the  supplier. 
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New  Informative  Material 


Update  selected  process  areas  to  provide  interpretation  of  practices  for 
organizations  with  respect  to  the  following  topics: 

•  Agile  methods 

•  Quality  attributes  (i.e.,  non  functional  requirements  or  “ilities”) 

•  Allocation  of  product  capabilities  to  release  increments 

•  Product  lines 

•  System  of  systems 

•  Architecture-centric  development  practices 

•  Technology  maturation 

•  Customer  satisfaction 


CMMI  VI  .3  and  Beyond 
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Software  System  Drivers 


Mission  Drivers 


If  function  were  all 
that  mattered,  any 
monolithic  software 
would  do,  ..but  other 
things  matter. . . 


The  important  quality  attributes  and  their  characterizations  are  key. 


'  •  Performance 

•  Availability 

•  Modifiability 

•  Interoperability 

•  Availability 

•  Security 

...  J 

t 

Business  Drivers 


analysis,  design,  development 


( 


Quality 
Attribute  Drivers 


X 


Software 

Architecture 


X 


Software  J 

_ I 


has  these  qualities 
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•  is  orchestrated 

•  has  a  plan 

•  knows  when  they 
deviate 

•  can  predict  the  impact 
of  changes 

•  measures  and  reports 
its  own  performance 

•  can  respond  quickly 


What  about  Agility? 

Agility  in  acquisition  and  devejopment,  as  with  operational  forces, 

requires  a  highly  disciplined  team. 

An  agile  team 
•  is  trained 
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VI  .3  Changes  to  High  Maturity  PAs 


Many  of  the  most  significant  changes  to  CMMI  models  as  part  of  Version 
1.3,  are  the  changes  to  the  high  maturity  process  areas  (CAR,  OPM,  OPP, 
and  QPM). 

These  process  areas  are  core  process  areas,  but  we’ve  focused  on  these 
four  over  the  others  because  of  their  significance  in  this  release. 
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High  Maturity  Changes  for  VI. 3 


Terminology  Confusion 

•  Common  Cause  (Statistical  versus  Quantitative  Techniques) 

•  Process  Models  and  Process  Modeling 

•  Business  Objectives 

•  Subprocesses 

Requirements  implied  versus  explicit/  Explanations  not  central  or  consistent 

•  Model/ Audit  Criteria/  Presentations  (Healthy  Ingredients)/  UCHMP 
Perceptions 

•  Customers  -  ML  5  is  expensive  -  no  better  than  3 

•  Industry  -  ML  5  is  NOT  RIGHT  for  every  business 
High  Maturity  in  ALL  constellations 

•  Examples  are  focused  on  Development 
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High  Maturity  Restructuring  for  VI  .3 


•  Insufficient  link  between  process  improvement,  business  objectives,  and 
performance 

•  Clarify  distinction  between  ML4  and  ML5 

•  Eliminate  GG4  and  GG5 

•  Make  CAR  more  relevant  for  organizational  benefit 
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Combined  OID  and  OPM  into  One  PA 


Performance  issues 
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Causal  Analysis  and  Resolution 


SG  1 :  Determine  Causes  of  Selected  Outcomes 
SP  1 .1  Select  Outcomes  for  Analysis 

SP  1 .2  Analyze  Causes 

SG  2:  Address  Causes  of  Selected  Outcomes 
SP  2.1  Implement  Action  Proposals 

SP  2.2  Evaluate  the  Effect  of  Implemented  Actions 
SP  2.3  Record  Causal  Analysis  Data 


Used  “outcomes”  instead  of  “defects 
and  problems.” 


Added  examples  for  service 
organizations  and  for  selecting 
outcomes  for  analysis. 


Added  subpractices  in  SP  1 .1  for 
defining  the  problem,  and  in  SP  2.2 
for  following  up  when  expected 
results  did  not  occur. 


Added  more  information  about  how 

PPMs  can  be  used. 


Added  emphasis  on  prevention  and 
reducing  recurrence. 
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Organizational  Performance  Management 


SG  1:  Manage  Business  Performance 

SP  1.1  Maintain  Business  Objectives 
SP  1 .2  Analyze  Process  Performance  Data 
SP  1 .3  Identify  Potential  Areas  for  Improvement 
SG  2:  Select  Improvements 


SP  2.1 

Elicit  Suggested  Improvements 

SP  2.2 

Analyze  Suggested  Improvements 

SP  2.3 

Validate  Improvements 

SP  2.4 

Select  and  Implement  Improvements  for 
Deployment 

:  Deploy  Improvements 

SP  3.1 

Plan  the  Deployment 

SP  3.2 

Manage  the  Deployment 

SP  3.3 

Evaluate  Improvement  Effects 

Renamed  the  PA  to  be 

Organizational  Performance 
Management  (OPM). 


Added  a  new  goal  about  managing 
business  performance  using 
statistical  and  other  quantitative 
techniques. 


Provided  more  information  about 
how  improvements  can  be  selected 
for  deployment. 


More  explicitly  described  and 
discussed  using  process 
performance  models. 

Clarified  that  not  all  improvement 
validations  include  piloting. 
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Organizational  Process  Performance 


SG  1:  Establish  Performance  Baselines  and  Models 

SP  1 .1  Establish  Quality  and  Process  Performance 
Objectives 

SP  1 .2  Select  Processes 

SP  1 .3  Establish  Process  Performance  Measures 

SP  1 .4  Analyze  Process  Performance  and  Establish 
Process  Performance  Baselines 

SP  1 .5  Establish  Process  Performance  Models 


Re-ordered  SPs,  moving  the  old  SP 
1.3  (Establish  Quality  and  Process 
Performance  Objectives)  to  SP  1.1 


Revised  SP  1 .4  to  include  process 
performance  analysis  and 
assessment  of  subprocess  stability. 


Revised  SP  1 .5  to  note  that  under 
certain  circumstances,  projects  may 
need  to  create  their  own  process 
performance  models. 


Clarified  the  relationship  of  OPP  to 
other  high  maturity  process  areas. 
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Quantitative  Project  Management 


SG  1 :  Prepare  for  Quantitative  Management 
SP  1.1  Establish  the  Project’s  Objectives 
SP  1 .2  Compose  the  Defined  Process 
SP  1 .3  Select  Subprocesses  and  Attributes 
SP  1 .4  Select  Measures  and  Analytic  Techniques 
SG  2:  Quantitatively  Manage  the  Project 

SP  2.1  Monitor  the  Performance  of  Selected 
Subprocesses 

SP  2.2  Manage  Project  Performance 

SP  2.3  Perform  Root  Cause  Analysis 


Restructured  QPM  so  that  SGI 
focuses  on  preparation  and  SG2 
focuses  on  managing  the  project. 


Added  guidance  about  using 
process  performance  baselines  and 
process  performance  models. 


Define  quantitative  management  in 
the  glossary  to  include  statistical 
management  and  use  that 
definition  for  use  of  the  terms 
throughout  QPM. 


Removed  the  practice  about 
applying  statistical  methods  to 
understand  variation  to  reduce  the 
over-emphasis  on  control  charts. 


Added  new  practices  about 
managing  performance  and 
performing  root  cause  analysis 
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CMMI  for  Development 
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CMMI  for  Development  Model 


cmmi-devIB 


‘  5 


Shared  PA  (SAM) 


T) 

CMMI 

Core  PAs 


Core  PAs  that  are 
present  in  all  CMMI 
models. 


CMMI-SVC 


CMMI-ACQ 
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Development-Specific  PAs 


CMMI-DEV  PAs  by  Maturity  Level 


Maturity  Level 

Process  Areas 

5 

Onfim^inn  Causal  Analysis  and  Resolution 

upumizmg  Organizational  Performance  Management 

4 

Quantitatively  Managed 

Organizational  Process  Performance 

Quantitative  Project  Management 

Decision  Analysis  and  Resolution 
Integrated  Project  Management 
Organizational  Process  Definition 
Organizational  Training 
Organizational  Process  Focus 

3  Defined  Product  Integration 

Requirements  Development 
Risk  Management 
Technical  Solution 
Validation 
Verification 

Configuration  Management 
Measurement  and  Analysis 
Project  Monitoring  and  Control 
Project  Planning 

Process  and  Product  Quality  Assurance 
Requirements  Management 
Supplier  Agreement  Management 


2  Managed 


For  the  VI  .3 
release,  there  were 
no  changes  that 
affected  the  DEV 
PAs’  positioning  by 
maturity  level. 
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CMMI-DEV  PAs  by  Category 


Process  Management 

Organizational  Innovation  and  Deployment  (OID) 
Organizational  Process  Definition  (OPD) 
Organizational  Process  Focus  (OPF) 
Organizational  Process  Performance  (OPP) 
Organizational  Training  (OT) 

Support 

Causal  Analysis  and  Resolution  (CAR) 
Configuration  Management  (CM) 

Decision  Analysis  and  Resolution  (DAR) 
Measurement  and  Analysis  (MA) 

Process  and  Product  Quality  Assurance  (PPQA) 


For  the  VI  .3  release,  REQM  was  moved  from 
“Engineering”  to  “Project  Management.” 


Project  Management 

Integrated  Project  Management  (IPM) 
Project  Monitoring  and  Control  (PMC) 
Project  Planning  (PP) 

Quantitative  Project  Management  (QPM) 
Requirements  Management  (REQM) 

Risk  Management  (RSKM) 

(+)  Supplier  Agreement  Management  (SAM) 

Engineering 

Product  Integration  (PI) 

Requirements  Development  (RD) 

Technical  Solution  (TS) 

Validation  (VAL) 

Verification  (VER) 
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Product  Integration 


SG  1 :  Prepare  for  Product  Integration 

SP  1.1  Establish  an  Integration  Strategy 

SP  1.2  Establish  the  Product  Integration  Environment 

SP  1.3  Establish  Product  Integration  Procedures  and 
Criteria 

SG  2:  Ensure  Interface  Compatibility 

SP  2.1  Review  Interface  Descriptions  for 
Completeness 

SP  2.2  Manage  Interfaces 

SG  3:  Assemble  Product  Components  and  Deliver  the 
Product 


Revised  the  purpose  statement  to 
ensure  proper  behavior  instead  of 
proper  function,  thereby  more 
explicitly  including  quality 
attributes  and  required 
functionality. 


Changed  emphasis  on  integration 
sequence  to  an  emphasis  on 

integration  strategy. 


Described  an  integration  strategy 
and  how  it  relates  to  an  integration 
sequence. 


SP  3.1  Confirm  Readiness  of  Product  Components 
for  Integration 

SP  3.2  Assemble  Product  Components 

SP  3.3  Evaluate  Assembled  Product  Components 

SP  3.4  Package  and  Deliver  the  Product  or  Product 
Component 
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Requirements  Development 


SG  1 :  Develop  Customer  Requirements 
SP1.1  Elicit  Needs 

SP  1 .2  Transform  Stakeholder  Needs  into  Customer 
Requirements 

SG  2:  Develop  Product  Requirements 

SP  2.1  Establish  Product  and  Product  Component 

Requirements 

SP  2.2  Allocate  Product  Component  Requirements 

SP  2.3  Identify  Interface  Requirements 

SG  3:  Analyze  and  Validate  Requirements 


SP1 .2  revised  to  add  that  customer 
requirements  should  be  prioritized 
based  on  their  criticality  to  the 
customer  and  other  stakeholders. 


Broadened  emphasis  from 
“operational  scenarios”  to  a  more 
balanced  “scenarios  (operational, 
sustainment,  and  development).” 

Added  a  focus  on  architectural 
requirements. 


SP  3.1 
SP  3.2 

SP  3.3 
SP  3.4 
SP  3.5 


Establish  Operational  Concepts  and  Scenarios 

Establish  a  Definition  of  Required  Functionality 
and  Quality  Attributes 

Analyze  Requirements 

Analyze  Requirements  to  Achieve  Balance 

Validate  Requirements 


Because  “Quality  attributes” 

needs  to  be  considered  in  addition  to 
“functionality,”  SG3  and  SP  3.2  were 
revised. 

Added  informative  material  that 
requirements  can  be  monitored 
through  development  based  on  their 

criticality  to  the  customer. 
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CMMI  and  Other  Models  and 

Standards 
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CMMI  Planned  Elements:  Multi-Model 


Improving  interfaces 
is  of  interest  to  both 
government  and 
industry.... 
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CMMI  and  the  People  CMM 


CMMI  (DEV,  ACQ,  SVC)  improves  the  capability  of  organizations’ 
processes  within  specific  domains. 

The  People  CMM  improves  the  capability  of  organizations’  workforces 
through  enhanced  management  and  human  capital.  (The  People  CMM 
defines  capability  as  the  level  of  knowledge,  skills,  and  process 
abilities  available  within  each  workforce  competency  of  the  organization  to 
build  its  products  or  deliver  its  services.) 


People  CMM 


Workforce  _  enables 
capability 


CMMI-DEV, 
-ACQ,  -SVC 


Process  —  improves 

Performance 

capability 
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Late  Discovery  of  System-Level  Problems 


Requirements 

Engineering 


System 
Design 


Software 

Architectural 

Design 


70%,  3.5% 
lx 


Component 
Software 
Design 

Sources:  v 

NIST  Planning  report  02-3,  The  Economic  Impacts  of  Inadequate 
Infrastructure  for  Software  Testing,  May  2002. 

D.  Galin,  Software  Quality  Assurance:  From  Theory  to 
Implementation,  Pearson/Addison-Wesley  (2004) 

B.W.  Boehm,  Software  Engineering  Economics,  Prentice  Hall  (1981) 


Vo,  3% 


10%,  50.5% 
16x 


20%,  16% 
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Code 

Development 


Where  faults  are  introduced 

Where  faults  are  found 

The  estimated  nominal  cost  for  fault  removal 
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Theory  of  Constraints  (TOC)  Definition 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


TOC  is  a  management  philosophy  introduced  by  Dr.  Eliyahu  M.  Goldratt  in  his 
1984  book  The  Goal,  that  is  geared  to  help  organizations  continually  achieve  their 
goal.  Based  upon  the  contention  that  any  manageable  system  is  limited  in 
achieving  more  of  its  goal  by  a  small  number  of  constraints  (&  that  there  is  always 
at  least  one).  The  TOC  process  seeks  to  identify  the  constraint  and  restructure  the 
rest  of  the  organization  around  it,  through  the  use  of  the  Five  Focusing  Steps. 

Key  Assumption 

The  underlying  assumption  of  TOC  is  that  organizations  can  be  measured  and 
controlled  by  variations  on  three  measures: 

■  Throughput  is  money  (or  goal  units)  generated  through  sales. 

■  Inventory  is  money  the  system  invests  in  order  to  sell  its  goods  and  services. 

■  Operating  Expense  is  all  the  money  the  system  spends  in  order  to  turn  inventory  into 
throughput. 


Standing  on  the  Shoulder’s  of  Giants 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


Theory  of  Constraints  International  Certification  Organization 

www.tocico.org 


Dr.  James  R.  Holt,  PE 

Professor 

Engineering  &  Technology  Management: 
Washington  State  University 
jholt@wsu.edu 

Home  Office:  (503)  669-6676 
Address:  429  SE  13th  Court 
Gresham  OR  97080-9361 
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The  Goal. 

It’s  not  Luck. 

Critical  Chain 

Necessary  but  not  Sufficient. 
The  Choice 
Isn't  it  obvious 


Its  Not  Luck,  Eli  Goldratt,  1997 

This  textbook  /novel  introduces  the  Thinking  Processes  and  shows  their  application 
of  the  Thinking  Process  in  many  different  environments 

The  Logical  Thinking  Process:  A  Systems  Approach  to  Complex 
Problem  Solving,  H.  William  Dettmer,  2007 

This  is  an  excellent  Theory  of  Constraints  Handbook 


Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 


Theory  of  Constraints  Basic  Facts 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Every  system  is  part  of  a  larger  system 

■  Every  system  has  sub-systems 

■  A  system  is  made  up  of  individual  elements  that  are 
linked  in  some  interdependent  fashion 

■  Within  any  complex  system  there  is  one  constraint  (or 
very  few)  of  several  possible  types 

■  Every  system  has  a  limiting  factor,  else  growing  systems  would 
soon  explode 

■  It  is  always  possible  to  find  the  constraint  and  exploit  it 
or  improve  it 


The  Five  Focusing  Steps 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


0.  What  is  the  Goal 

0.5  How  to  Measure  Progress  toward  Goal 

1.  Find  the  System  Constraint 

2.  Decide  How  to  Exploit  the  Constraint 

3.  Subordinate  everything  else  to  the  Constraint 

4.  Elevate  the  constraint 

5.  If  the  constraint  moves,  start  over  at  Step  1 


Constraints  Management 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  In  physical  systems  (machines,  production, 
distribution, ... )  the  constraint  is  relatively  easy  to  find 

■  In  non-physical  systems  (social  systems,  service 
organizations,  interactions,  relationships,  self- 
governing,  not-for-profit,  creative  groups,  ... )  it  is 
harder  to  find 

■  While  we  can  often  change  our  minds  faster  than 
changing  equipment,  getting  EVERYONE  to  change 
their  minds  at  the  same  time,  is  pretty  hard 
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Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 


Intermediate 
Objectives  Map 


What  is  the  GOAL  and  what 
are  the  steps  to  reach  it? 


Transition  Tree 


Objective 

Intermediate  Effects 
Specific  Actions 


Negative  Branch  Reservation 
Prerequisite  Tree  (PRT) 
Transition  Tree  (TT) 


A 


TOC  Thinking  Processes:  Constructs 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Evaporating  Clouds 
(Conflict,  3  clouds,  CCC) 
Assumptions 

Undesirable  Events  (UDEs) 
Current  Reality  Tree  (CRT) 
Injections  on  CCC 
Desirable  Events 
Future  Reality  Tree  (FRT) 


Goal 

t 

Critical  success  factors 

t 

Necessary  Conditions 


Evaporating  Cloud 

(Conflict  Resolution  Diagram) 

- .  X- 

Objective 

t 

Requirements 

t 

Prerequisites 

How  to 


Prerequisite  Tree 


Objective  (Injection) 

t 

Obstacles 

Intermediate 

Objectives 


What  to 


CAUSE 

the 

change  ? 

\  change 
\  TO? 

Future  Reality  Tree 

/ 

Desired  Effects 

t 

1 

Intermediate  Effects 

t 

Injections  -|q 

Source:  The  Logical  Thinking  Process,  H.  William  Dettmer,  2007 


Different  Logic  Patterns 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


Logic:  There  are  two  main  types  of  logic:  Sufficiency-Based  logic  and 
Necessary-Based  Logic 


I  have  a  Dog 


I  have  an  Animal 


■  With  sufficiency  based  logic,  we  read  in  the  direction  of  the  arrow  and  say  IF  A, 
THEN  B,  we  mean,  "It  is  sufficient  that  if  A  exists,  then  B  exists."  IF  I  have  a 
dog,  THEN  I  have  an  animal.  Having  a  dog  is  sufficient  to  satisfy  the  conclusion 
that  I  have  an  animal.  This  is  common  sense 


■  With  necessary  based  logic,  we  read  against  the  arrow  and  say,  In  Order  To  C, 

I  Must  Have  D.  In  Order  To  make  a  fire,  I  Must  Have  combustible  material.  This 
is  also  common  sense.  Can  you  have  a  fire  without  combustible  material? 


A  Fire 


< 


/ - \ 

Combustible  Material 


■  These  two  different  types  of  logic  are  often  confused,  but  shouldn't  be.  One  is 
forward  thinking,  the  other  is  backwards  thinking.  They  are  Different 


■  IN  ORDER  TO  have  an  animal  I  MUST  HAVE  a  dog.  Is  that  common  sense? 

No,  I  could  have  a  cat.  IF  I  have  combustible  material  THEN  I  have  a  fire.  Is  that 
common  sense?  No,  there  are  other  things  necessary  to  have  a  fire.  The 
categories  of  legitimate  reservation  make  sure  the  logic  is  solid 


source:  Washington  State  University-2010 
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Alternating  Logic  Types 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  The  Theory  of  Constraints  Thinking  Processes  use  both  necessary  and 
sufficient  logic.  In  examining  the  logic  tools  closely,  you  see  the  tools  alternate 
the  logic  types.  The  CRT,  FRT  and  TT  use  sufficiency  based  logic.  The  EC  and 
PRT  use  necessary  based  logic.  This  clever  alternating  logic  causes  the  final 
conclusion  to  not  only  be  necessary  but  also  sufficient--the  solution  will  be  the 
minimum  set  of  required  conditions  to  be  sufficient  to  assure  success. 

■  These  two,  common  logic  types  each  discover  causality  in  different  ways.  By 
alternating  them,  we  clarify  and  better  understand.  We  recognize  that  all  dogs 
are  animals.  And,  we  need  heat  and  oxygen  besides  combustible  material  to 
create  a  fire.  The  use  of  both  makes  our  thinking  process  robust,  forgiving  and 
reliable.  Using  them  both  will  help  you  discover  new  knowledge. 

■  Some  people  claim  necessary  based  logic  is  right  brained  and  sufficiency  based 
logic  is  left  brained.  Maybe  getting  both  sides  of  the  brain  to  work  together  will 
be  a  boon!  Anyone  can  learn  necessary  based  logic.  It  just  takes  practice. 


source:  Washington  State  University-2010 
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Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 
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Evaporating  Clouds  are  Necessary-Based  Logic 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Necessary  based  logic  is  the  basis  for  the  Evaporating  Cloud  (Conflict  Resolution 
Diagram  as  Dettmer  calls  it).  What  are  the  necessary  conditions  to  achieve  the  goals  we 
desire?  What  are  the  prerequisites  for  the  necessary  conditions? 

■  The  Evaporating  Cloud  is  used  to  surface  intuition,  generally  accepted  assumptions, 
unspoken  understanding,  and  established  rules. 

■  These  areas  (assumptions  behind  the  necessary  logic  arrows)  then  become  targets  for  challenge  / 
change/  improvement. 

■  The  traditional  assumptions  limit  the  range  of  solutions  to  our  problems. 

■  If  the  traditional  assumptions  can  be  made  invalid,  then  we  have  found  a  place  where  the 
necessary  conditions  no  longer  hold. 

■  We  call  such  a  discovery,  a  breakthrough  injection. 

■  The  Evaporating  Cloud  is  the  most  easily  used  logic  tool.  It  can  be  used  everyday  to 
develop  understanding,  facilitate  communication  and  resolve  every-day  conflict.  Students 
of  the  Thinking  Process  should  learn  to  create  Evaporating  Clouds  in  minutes.  It  is  most 
better  to  resolve  conflicts  quickly,  before  they  become  chronic 


source:  Washington  State  University-2010 
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Here  is  an  Evaporating  Cloud  Worksheet 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


Story  line: 


source:  Washington  State  University-2010 
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Guidelines  for  Creating  and  Evaporating  Cloud 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


As  soon  as  you  recognize  there  is  no  room  for  an  acceptable  compromise  (there  is  a  breakdown  in 
communication,  the  other  side  is  becoming  stubborn  and  illogical  or  our  relationship  is  in  jeopardy) 
take  a  break.  During  the  break: 

1.  Write  down  WHAT  I  WANT 

You  know  this.  You  keep  repeating  it  over  and  over  out  loud  and  in  your  thoughts.  (Usually,  this  is  an 
action  or  result  of  an  action) 

2.  Write  down  WHAT  THE  OTHER  SIDE  WANTS 

You  know  this.  They  keep  repeating  it  over  and  over  trying  to  convince  you. 

3.  Write  down  WHAT  NEED  I  AM  TRYING  TO  SATISFY  (B) 

You  may  have  to  stop  and  think  hard  here.  Why  is  it  you  NEED  the  thing  you  want? 

4.  Write  down  WHAT  NEED  THE  OTHER  SIDE  IS  TRYING  TO  SATISFY  (C) 

If  you  don't  know,  you  haven't  been  listening  to  the  other  side.  Think!  What  was  their  justification  for 
their  desires?  That  was  their  need? 

5.  Write  down  OUR  COMMON  OBJECTIVE  (A) 

Yes,  you  have  one.  You  are  still  talking!  What  is  the  reason  you  are  still  talking  to  this  hard  headed, 
obnoxious,  uncooperative  person  on  the  other  side?  What  do  you  both  want  to  have?  What  goal  do  you 
both  need  to  satisfy?  Why  is  it  you  are  trying  so  hard  to  convince  the  other  side  to  cooperate? 


source:  Washington  State  University-2010 
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Creating  the  Evaporating  Cloud 


Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


A. 

(5.  What  is  our 
common  Goal 
or  Objective?) 


B. 

(3.  What  need 
am  I  trying  to 
satisfy?) 


'  c. 

I  D‘  1 

(4.  What  need 

(2.  What 

is  the  other 

◄ - 

does  the 

side  trying  to 

other  side 

v  satisfy?)  , 

V  want?)  ) 

In  order  to  ... 

Objective 

source:  Washington  State  University-2010 


We  need  to  ... 
Requirement 


In  order  to. 


We  need  to  ... 
Prerequisite 
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Focusing  Lean  Support  Group 
Core  Conflict  Cloud 


Story  line:  The  company  has  limited  resources  to  perform  “Process  Improvement”  and  support  internal  customer  needs  for  Change 
and  Improvement.  These  “change  agents”  are  tasked  with  working  with  Employee  Involvement  Teams  to  create  a  culture  of 
process  improvement.  There  is  an  effort  underway  to  explore  TOC  as  an  improvement  tool. 


Combine  LEAN  and  TOC  for  fast,  surgical  results! 


Assumptions 

1.  There  is  pressure  to  try  something  better  than  what  we  have  been  doing 

2.  Our  customer  has  changing  requirements 

3.  The  customer  wants  products  faster  and  cheaper 

4.  Difficult  to  schedule  today  what  is  going  to  occur  more  than  6  months  in  the 
future 

5.  Our  project  requirement  are  not  all  the  same 


Train  TOC  folks  on  how  they  can  benefit  from  LEAN  (to  improve  the 
difficult  processes  they  find  -  in  CCPM  for  sure  -  and  other 
bottleneck  areas). 

Assumptions 

1.  Our  current  date  driven  scheduling  causes  us  to  have  to  do  project  rescheduling 

2.  There  is  pressure  to  try  something  better  than  what  we  have  been  doing 

3.  we  need  to  improve  our  processes 

4.  TOC  takes  in  to  account  delays  and  resources  availability 

5.  TOC  is  a  good  way  to  create  process  that  are  adaptive _ 


Be. 

3  Do  things  that  deliver 
significant  results  quickly 


Assumptions 

1 .  We  don’t  know  where  to  focus  resources 

Implementing  both  concepts  causes  confusion 
Lean  and  El  don’t  always  deliver  desired  results 
Not  everyone  knows  TOC 


Dc.’ 

1.  Take  radical,  unproven  (to  me), 
experimental  procedures 


Use  the  Thinking  Tools  Process  to  establish 
priorities  for  Process  Improvement  Projects 


Focus  resources  on  where  the  highest  savings  are  (largest 
improvement  can  be  made  -  that  contribute  to  Throughput  immediately 
(or  accelerate  project  management  processes) 


V 


Assumptions 

1 .  We  need  to  improve  our  processes 

2.  Stable  processes  makes  jobs  easier  to  manage 

3.  We  know  how  to  control  stable  process 

4.  If  we  change  we  might  create  an  unstable  process 


f  ^ 

c 

\ 

Cc. 

4.  Do  mature,  correct  things  that  have 

* - 

Dc. 

proven  to  be  very  effective  over  time 

1  ¥ 

t 

2.  Use  the  best  LEAN  tools  and 
techniques,  El  (Employee  Involvement) 

Assumptions 

1 

J 

There  is  a  method  to  create  and  maintain  stable 
balanced  processes  (We  become  a  fast,  reliable 
source  of  expert  knowledge  and  projects.  We  can 
pick  and  choose  which,  of  many,  contracts  we  will 
pursue.  We  are  profitable.  We  can  grow  our 
capacity  at  a  rate  we  choose. 


1.  Toyota  like  methods  have  been  up  to  this  point  the  company  accepted  method 
for  continuous  improvement. 

2.  Toyota  is  considered  to  be  successful  with  their  implementation  of  Lean 
Processes 

3.  The  accepted  site  improvement  methodology  has  been  El 

4.  It  is  easier  to  stay  with  the  current  culture 

5.  El  teams  are  easy  to  measure  and  control _ 


Train  Lean  Facilitators  on 
TOC 


Create  meaningful  improvement 
metrics  associated  with  Increasing 
Value  and  decreasing  Time 


Find  other  ways  to  implement  Employee  Involvement 
(remove  the  obstacles  to  being  fast  and  let  the  work 
itself  be  the  reward) 


In  order  to  ... 
Objective 


We  need  to  ... 
Requirement 


In  order  to... 


We  need  to  ... 
Prerequisite 


CLRs 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


CLRs  -  Categories  of  Legitimate  Reservation 

■  Are  tools  to  rigorously  evaluate  and  critique  (scrutinize)  logic 
statements  in  a  non-threatening  manner 

■  Ensure  the  logic  is  solid 

■  Help  solidify  intuition  or  emotion  into  solid  cause-effect 
relationships 

■  Force  the  discovery  new  knowledge  to  correct  logical  errors 

■  The  Categories  of  Legitimate  Reservation  are  the  tools  to 
validate  or  invalidate  any  argument.  They  are  very  powerful 
when  applied  in  a  systematic  way.  They  are  the  tools  to 
discover  the  underlying  assumptions. 


source:  Washington  State  University-2010 
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Categories  of  Legitimate  Reservation 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


source:  Washington  State  University-2010 
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Scrutiny! 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  What  is  Scrutiny? 

■  We  want  to  examine  each  logical  arrow  according  to  eight  logic 
rules  (CLRs). 

■  Taken  together  with  your  intuition,  they  are  sufficient  to 
demonstrate  the  truth  of  your  logical  connections. 

■  Or,  they  will  surface  legitimate  reservations  about  the  logic. 

■  Anytime  a  reservation  is  surfaced  and  addressed,  additional 
knowledge  is  gained.  DISCOVERY! 

■  Use  Scrutiny  to  gain  deeper  understanding/discovery. 

■  Use  it  on  yourself  and  with  others.  Encourage  them  to  scrutinize 
you! 


source:  Washington  State  University-2010 
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Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 
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Creating  a  Current  Reality  Tree  from  UDE 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  UDEs  -  Undesirable  Effects 

■  Let’s  approach  these  UDEs  and  the  whole  world  of  Team 
Management  by  taking  three  different  situations  (one  UDE 
clouds)  and  combining  them. 

■  Development  (or  Design)  /  Business  /  Education 

■  Other  areas  would  have  worked  just  as  well;  these  are 
good  enough 


source:  Washington  State  University-2010 
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UDEs  of  Lean  Support  Group 


Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  The  majority  of  BDS  leadership  has  the  perception  that  lean  does  not  generate  results. 

■  ‘Employee  Involvement  has  not  produced  the  results  desired 

■  ‘Lean  has  not  produced  the  results  desired 

■  ‘Leadership  is  hesitant  to  apply  lean  to  a  business  problem. 

■  ‘Lean  results  do  not  show  up  in  EAC 

■  ‘Leadership  feels  leading  Lean  is  not  a  valuable  way  to  spend  their  time. 

■  ‘Leadership  fears  using  Lean  because  of  fear  of  not  being  able  to  proved  the  expected  improved  financials. 

■  ‘Lean  facilitators  are  most  comfortable  with  soft  skills  facilitation  /  training  and  not  driving  teams  to  achieve  significant 
improvement  fast 

■  ‘Lean  facilitators  are  not  comfortable  not  driving  teams  to  achieve  significant  improvement  fast. 

■  ‘Lean  facilitators  are  conflict  adverse.. 

■  ‘Lean  facilitators  become  ineffective  when  having  to  lead  real  change. 

■  ‘People  are  confused  at  which  measure  to  follow 

■  ‘People  are  frustrated 

■  ‘People  are  not  happy  (satisfied)  with  their  work 

■  ‘The  site  is  not  performing  as  well  as  it  should 

■  *  Lean  facilitators  are  not  sure  which  metrics  to  track  and  follow 

■  ‘There  is  general  confusion  about  what  the  goals  and  objectives  are 

■  ‘There  is  a  general  lack  of  focus  and  direction 

■  *  Lean  facilitators  are  not  supporting  all  the  needs  of  the  customers 

■  ‘Employee  Involvement  is  not  capturing  as  much  savings  as  possible 

■  ‘There  is  conflicting  direction  from  matrix  chain  of  command 

■  ‘There  is  lack  of  emphasis  on  improvements  that  impact  the  bottom  line 

■  ‘There  is  confusion  on  which  tools  and  techniques  to  use 
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Does  this  Conflict  seem  to  CAUSE  all  the  UDEs? 


Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


Some  team  members 
complain 


Team  seems  out-of- 
control 


Teams  cost 
more 


source:  Washington  State  University-2010 
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Behind  Every  Arrow,  There  is  an  Assumption 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  The  assumption  in  Necessary  Based  logic  was: 

“In  order  to  Eat,  I  must  have  Money  Because  I  pay  for  my 
food.” 


■  The  assumption  in  Sufficiency  Based  logic  is  the  same  but  is 
structured  slightly  differently: 

“If  I  want  to  eat,  And  I  pay  for  my  food,  Then  I  need  money.” 
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Some  Assumptions  Behind  the  Arrows  of  the  Core 
Conflict 


Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


Effective  teams 
have  a  high 
degree  of  synergy 


V 


Any  individual  on  the  team  can 
cause  team  failure 


B 

D 

\ 

Have  a  High 

Base  Recognition 

Degree  of  Group 

on  Team 

Cooperation 

) 

V _ 

Performance 

J 

Converting  the  Evaporating  Cloud  to  Current 
Reality  Tree 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  The  next  few  steps  are  Dramatic  Ones: 

■  Stand  the  Cloud  on  its  point. 

■  Turn  around  the  arrows 

■  Adjust  the  wording  to  make  smooth 
“If...  Then  ...”  logic 

■  Add  the  Assumptions  with  the  ‘And’  Connectors 

■  The  result  is  the  start  to  the  Current  Reality  Tree! 


source:  Washington  State  University-2010 
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Start  with  the  Conflict  Cloud 
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A 

Have  a 
Successful 
Team 


N 


B 

Have  a  High 
Degree  of 
Group 
Cooperation 

■CB 

Encourage 

Individual 

Contribution 


source:  Washington  State  University-2010 


Base  Recognition 
on  Team 
Performance 


Not  D 

Base  Recognition  on 
Individual 
Performance 
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source:  Washington  State  University-2010 
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Turn  the  Arrows  Around 
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\ 

Base  rewards  on 
Individual 
performance 


Have  a  High 
degree  of  group 
coooeration 


Encourage 

individual 

contribution 


Base  rewards  on 
team 

performance 


source:  Washington  State  University-2010 
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source:  Washington  State  University-2010 
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Add  in  at  least  one  Assumption  for  each  arrow 
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There  is  pressure  to 
base  rewards  on  team 
performance 


Any  individual 
on  the  team  can 
cause  team 
failure 


There  is  pressure  to  base 
rewards  on  individual 


performance 


We  need  a  high  degree 
of  group  cooperation 


We  need  individual 
contributions 


Individuals 
respond  to 
individual 
rewards 


Effective  teams 
have  a  high  degree 
of  synergy 


We  want  to  have  a 
Successful  Team 


The  team’s  work 
is  mostly 
individual  effort 


source:  Washington  State  University-2010 
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Check  the  Logic  using  the  Categories  of  Legitimate 
Reservation 
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Numbers  added  as 
labels  to  entities  to 
facilitate  discussion 


145  There  is  pressure 
to  base  rewards  on 
team  performance 


125  Any  individual  on  the 
team  can  cause  team 
failure 


130  We  need  a  high 
degree  of  group 
cooperation 


150  There  is  pressure  to 
base  rewards  on  individual 
performance 


135  We  need 

\ 

individual 

V 

contributions 

) 

140  Individuals 
respond  to 
individual 
rewards 


110  Effective  teams  have  a 
high  degree  of  synergy 


100  We  want  to 
have  a  Successful 
Team 


120  The  team’s  work  is  mostly 
individual  effort 


source:  Washington  State  University-2010 
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Connect  the  Core  Conflict  to  the  UDES  for  the  CRT 
foundation 
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source:  Washington  State  University-2010 
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Starting  Groups  and  Connect 
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UDE  i 


source:  Washington  State  University-2010 


Do  some  UDEs  seem 
related? 

One  follow  the 
other? 

Group  them! 
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Simple,  Temporary  Connections 
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UDE 


...r 


UDE 


UDE 


source:  Washington  State  University-2010 
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Look  for  the  Result  of  the  Conflict 
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What  Happens  because  of  the  Conflict  and  D/D’? 
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source:  Washington  State  University-2010 
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Scrutinize  the  Entire  Tree  Using  the  CLR 
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source:  Washington  State  University-2010 
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Re-Evaluate  the  UDES-Find  New  Ones/Delete  Old  Ones 
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an  UDE) 


source:  Washington  State  University-2010 
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Look  for  Feedback  Loops 
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source:  Washington  State  University-2010 
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Look  for  Policy,  Measurement,  Behavior  Issues 
(add  them  if  they  are  not  there) 
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source:  Washington  State  University-2010 
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The  Cone  Shape  Then  Becon 
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1 

r 

NOT  A 

■  r 

.  i_  i 

NOT  C 


NOT  B 


source:  Washington  State  University-2010 


■  The  addition  to  the  Base 
divides  the  CRT  quickly 

■  Builds  upwards  quickly 

■  The  Regions  help  show 
what  is  missing 

■  Allows  a  Thinking 
Process  person  to  view 
the  CRT  and  Scrutinize 
Quickly 
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Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 
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■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 


46 


The  Future  Reality  Tree 
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■  The  FRT  is  NOT  detailed.  It’s  a  HIGHer  order  of  abstraction 

■  Its  purpose  is  to  affirm  direction  and  build  confidence 

■  Don’t  get  caught  up  in  too  much  minutia!  Make  it  High  Level  & 
Logically  Sufficient 

■  The  PRT  and  TT  will  do  the  detail  work  later. 

■  The  Core  Conflict  already  raised  your  thinking  to  a  higher  level  of 
abstraction  from  the  UDES.  Let  the  FRT  stay  near  that  level 

■  Still,  you  need  enough  detail  to  achieve  the  DEs 


source:  Washington  State  University-2010 
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Convert  UDEs  to  DEs 
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Convert  the  Undesirable  Entities  into  a  Positive  Desirable 
Effect. 

UDE:  Behind  the  Scenes 


UDE 


DE 


UDE 


DE 


UDE 


DE 


UDE 


DE 


maneuvering  (politics)  is  common 

DE:  People  are  very  cooperative 

UDE:  People  are  frustrated 

DE:  People  are  happy  on  the  job 

UDE:  Many  people  are  valued 
some  what  arbitrarily 

DE:  People  know  their 
contribution  will  be  appreciated 


source:  Washington  State  University-2010 
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Reaching  our  Future 
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DE 


DE 


Strategic  Injection 
(Silver  Bullet-Silver  Wedge) 
Super  Injection! 


We  want  to 
create  our 
Desired  Effects 
(opposites  of 
our  UDEs) 

Will  the  Silver 
Bullet  get  us 
there? 


source:  Washington  State  University-2010 
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Finding  Breakthroughs  (the  next  step) 
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■  Breakthrough  Solutions  are  Silver  Bullets  (or  at 

least  Silver  Wedges,  powerful  wedges  that  only 
need  a  crack  in  the  wall  to  allow  us  to  “work  our  way 
through”). 

■  How  do  we  find  them? 

■  By  Observation 

■  By  Challenge 

■  By  Reference  Environment 

■  The  best  are  often  The  Impossible  Ones 

■  If  you  can’t  find  one,  Just  do  the  Opposite  of  the  Rest 
of  the  World. 


source:  Washington  State  University-2010 
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■  The  Super  Injection,  may  represent  the  strategic 
direction. 

■  In  order  to  build  a  solid  FRT,  it  may  need  to  be  broken 
into  it’s  component  parts.  The  definition  of  the 
component  parts  then  assist  in  the  development  of  the 


FRT. 


More  Specific 
Injection 


Mile  Stone 
Injection 


Understandable 


Condition  Iciear  Directiori 


Strategic  Injection 
(Silver  Bullet-Silver  Wedge) 


source:  Washington  State  University-2010 
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The  Resulting  Future  Reality  Tree 
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The  FRT  gives  the 
road  map  that 
confirms  we  are  on 
the  right  track. 

It  shows  our 
injection(s)  can 
succeed. 

It  gives  confidence 
and  direction  in 
moving  into  the 
future. 


source:  Washington  State  University-2010 
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162.  SO:  A  management  system  is  in  place  to 
successfully  perform  Process  Improvements,  add 
value  and  meet  cost,  schedule  and  quality 
expectations 


Reality  Tree  -  Focused  Lean  Resources 

164.  SO:  The  Site  meets  it’s  cost,  schedule 
[and  quality  performance  expectations 


166.  SO:  The  site  consistently 
meets  it’s  profit  goals 


Strategic 

objective 


128  TO:  Process  Improvement 
projects  are  completed  and  support 
Site  goals 


118  DE  The  needs  of 
the  customers  are 

104  DE  Process  Improvements 
will  impact  the  bottom  line 

being  supported. 

102  DE  Lean 
Facilitators  will 
provide  desired 
results 


Tactical  objective 


190  DE  Lean  Facilitators 
will  be  more  effective. 


160  DE  People  are 
happy  (satisfied). 


170  DE  Lean 
Facilitators  don’t 
switch  from  project  to 
[project 


180  DE  Lean  Facilitators  know 
which  Process  Improvement 
rojects  to  focus  on 


60  INJ;  Process 
Improvements  are 
prioritized  according  to 
their  potential  savings 


80  INJ:  Establish  priorities  of 
Process  Improvement  projects 


.Combine  Lean  and 
rOC  for  fast,  suiflical  results 


70  INJ:  Use  TOC  5  Focusing 
steps  to  Prioritize  Process 
Improvement  Pxojects 


jj^ical  results 


10  SINJ  Improve  our  processes  by  using  tools  from  the  Theory  Of  Constraints  (5  focusing 
steps,  WIP,  priorities)  to  focus  Lean  resources. 


Applying  Theory  of  Constraints  Tools  to  Focus  Lean 
Development 

Boeing  Defense,  Space  &  Security  |  Wichita  Engineering 


■  Introduction  to  Theory  of  Constraints 

■  How  to  use  the  Thinking  Process  Tools 

■  What  to  Change 

-  Evaporating  Clouds  to  identify  conflicts 

-  Current  Reality  Tree 

■  What  to  Change  to 

-  Strategic  Future  Reality  Tree 

■  How  to  cause  the  Change 

-  Prerequisite  Trees  and  Transition  Trees 

-  Strategy  and  Tactics  Tree 
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The  Prerequisite  Tree 
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The  purpose  of  the  Prerequisite  Tree  is  to  surface  potential  obstacles  (OBS)  to  be  overcome 
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Pre-Requisite  Tree 


Super  INJ  Improve  our  processes  by  using  tools  from  the  Theory  Of  Constraints  (5  focusing  steps,  WIP, 
_ priorities)  to  focusj-ean  resources. _ 


INJ  Use  TOC  5  Focusing  steps  to  Prioritize  Process 
Improverr^ent^Projects 


The  capacity  of  the  Lean%|dent|fy  constraints  in^^t 


Facilitators  working 
Process  Improvement 
projects  is  unknown 


processes.  Then  subordinate  all  other 
Process  Improvement  projects  . 


Process  with 
constraints  are  not 
identified  or 
managed 


identified  and  managed  the  priority  Process 
Improvement  projects 


Lean  Facilitators  and 
Management  don’t 
know  about  the 
benefits  of  TOC 


Limit  the  number  of  Lean  Projects 
being  worked  at  any  time 


Support  the  Lean 
Facilitators  working  the 
priority  Process 
Improvemenpjrojects 


The  current  environment 
of  pushing  more  projects 
on  to  the  Lean  Resources 
than  they  can  handle 

'Bad  Multi- 
tasking  is 
allowed  and 
even 

^encouraged 


Release  Process  Improvement 
'  projects  work  to  the  Lean  Resources 
.using  a  Drum  Buffer  Rope 
methodology . 


Eliminate  bad  multitasking  and  ensure 
management  (and  everyone)  supports 
the  no  bad  multitasking  policy. 


INJ  Establish  Clear  priorities  for 
Process  Improvement  projects 


There  are  too  many 
interruptions,  such  as 
managers  wanting  their  pet 
projects  worked. 


The  priorities  are  not  clear,  the  Lean 
Resources  don’t  know  whether  they 
are  working  on  the  most  important 
^Process  Improvement  projects. 


Ensure  the  work  rules  are  followed. 


Devise  rules  and  policies  s  that 
handles  the  requests  that 
comes  into  the  Lean  support 
group. 


From  the  PRT  to  the  Transition  Tree 
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The  Transition  Tree 
CAUSES  the 
Intermediate  Objectives 
to  exist  so  that  the 
Injection  will  be 
achieved. 

When  all  Tactical 
Objectives  exist,  the 
FRT  becomes  our  new 
Current  Reality!!!! 


200  Work  is  scheduled  in  order  of 
priority  without  resource  contention. 
Work  is  staggered  to  avoid  conflicting 
resources 


N 


We  start  all 
work  as  soon 
as  possible^ 


We  control 
when  work  starts 


Our  work  is  ^ 
too  complex 
to  schedule 


e  can  schedule  the 
critical  few 
esources  accurately 


Everything  has 
ne  priorit 

_ /l 


Managers 
prioritize  work 

/  _ 

don’t  knovl^ 
f  about  conflicts 

%l,ntM  they  happen 

We  know  where 
limited  resources 
are  used 
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The  Transition  Tree  (TT)  follows  a  simple  format  to  achieve: 

The  Actions  that  Cause  the  Change 
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New  Reality  (changed 
reality)  is  achieved. 


There  is  a  simple 
Five  Cell  Structure 


Facts  of  life  that 
indicate  the 
action  will  be 
sufficient  to 
achieve  the  new 
reality 


The  Need 
for  Action 
(What  is 
needed) 


My  Action  (Emphasis 
on  the  action  I  take  to 
cause  someone  (or 
something)  to  meet  the 
need  and  change 
reality) 


Facts  associated  with 
Current  Reality  that  lead 
to  the  need  for  change. 


The  Current 
(previous) 
Reality 
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The  Transition  Tree  Fills  Logic  Below  and  Within  the 
Gap  between  the  Intermediate  Objectives  of  the  PRT 
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Injection 


Intermediate 
QfrjgOtive_ 


Need 

JWk 

.Qf  life. 


Facts 


of  lifeJ 


Facts  \ 
nf  life/ 

Intermediate 

Objective 

'\  ^ 

Facts 
.of  life. 

Need 

Injection 
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i  Improvement  projects  , 

1 1  >  %j?  t1  ixa  uy^viy  *.  *  %«i.t  i  ^6  S 


The  Lean  group  has 
agreed  to  use  the  new 
work  rules  and  measures 
in  all  Pr< 


80  INJ:  Establish  priorities 
of  Process  lmnmvement 
We  need  to 
use  the  new 
prioritization 
rules  and 


measures  / 
ecRtom  iui^s  for 


rA 


IThe  groups  priorities  are 
not  clear,  we  don’t  always 
know  whether  we  are 
working  ooJtie  most 
i  important  projects 


in  the  way  we  will  complete 
the  Process-Improvement 
projects 

interruptions,  with 
different  Managers 
wanting  different 
projects  worked  r 


prio 

all  the  types  of 
Process  Improvement 


Ensure  the  work 
rules  account  for 
all  types  of 
Process 
Improvement 
Projects 


and  measures  to 
control  the  work  and 
the  Lean  Support 


many  managers  desires 


We  have  cgfregories  for 


Process  Improvements 
identified  and  know  their 


We  need  to 
focus  our 
Lean 


improvement  project 

priorities  are  not 
established: - 


Process 
Improvement  project 
throughput  measures 


Devise  work  rules 
/to  handle  the 
projects  that  comes 
into  the  Lean 
Support  group 


Identify  the  categories 
and  kinds  of  Process 
Improvement  projects 
that  the  Lean  Support 
group  does  &  their 
_ priority _ 


■cncn 


I UOC33 


V. 


Improvement  project  is 
put  into  the  queue  with 
no  priority 


We  Need  to  know 
many  projects  the 
Lean  team  can 


|  Fact  | 

1  Need  I 

|  Action  | 

|  Fact  | 

|  New^Reality 

l 

- - - 

|  Fact  | 

[  Need 

J  |  Action  | 

on  all  the  Process 
Improvement  Projects  at 
once 


The  Lean  Facilitators 
are  always  busy 


We  understand  the 
Leanteams  RAA and 
can  measure  their 

thr°mhput  


Identify  Process 
Improvement  Projects 
as  throughput  to  the 
Lean  support  group 


-**«- 


We  have  a  lot  of  Process 
Improvement  Projects  to 
do 


value  of  each  Process 
Improvement  project 


Determine  Lean 
Facilitators  RAA 


Lean  Facilitators  are 
asked  to  do  multiple 
jobs 


Questions 
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sering 
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Focusing  Lean  Support  Group 
Core  Conflict  Cloud 


Story  line:  The  company  has  limited  resources  to  perform  “Process  Improvement”  and  support  internal  customer  needs  for  Change 
and  Improvement.  These  “change  agents”  are  tasked  with  working  with  Employee  Involvement  Teams  to  create  a  culture  of 
process  improvement.  There  is  an  effort  underway  to  explore  TOC  as  an  improvement  tool. 
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Combine  LEAN  and  TOC  for  fast,  surgical  results! 


Assumptions 

1.  There  is  pressure  to  try  something  better  than  what  we  have  been  doing 

2.  Our  customer  has  changing  requirements 

3.  The  customer  wants  products  faster  and  cheaper 

4.  Difficult  to  schedule  today  what  is  going  to  occur  more  than  6  months  in  the 
future 

5.  Our  project  requirement  are  not  all  the  same 


Train  TOC  folks  on  how  they  can  benefit  from  LEAN  (to  improve  the 
difficult  processes  they  find  -  in  CCPM  for  sure  -  and  other 
bottleneck  areas). 

Assumptions  - 

1.  Our  current  date  driven  scheduling  causes  us  to  have  to  do  project  rescheduling 

2.  There  is  pressure  to  try  something  better  than  what  we  have  been  doing 

3.  we  need  to  improve  our  processes 

4.  TOC  takes  in  to  account  delays  and  resources  availability 

5.  TOC  is  a  good  way  to  create  process  that  are  adaptive _ 


Be. 

3  Do  things  that  deliver 
significant  results  quickly 


Assumptions 

1 .  We  don’t  know  where  to  focus  resources 

Implementing  both  concepts  causes  confusion 
Lean  and  El  don’t  always  deliver  desired  results 
Not  everyone  knows  TOC 


Dc.’ 

1.  Take  radical,  unproven  (to  me), 
experimental  procedures 


Use  the  Thinking  Tools  Process  to  establish 
priorities  for  Process  Improvement  Projects 


Focus  resources  on  where  the  highest  savings  are  (largest 
improvement  can  be  made  -  that  contribute  to  Throughput  immediately 
(or  accelerate  project  management  processes) 

V 


Assumptions 

1 .  We  need  to  improve  our  processes 

2.  Stable  processes  makes  jobs  easier  to  manage 

3.  We  know  how  to  control  stable  process 

4.  If  we  change  we  might  create  an  unstable  process 

There  is  a  method  to  create  and  maintain  stable 
balanced  processes  (We  become  a  fast,  reliable 
source  of  expert  knowledge  and  projects.  We  can 
pick  and  choose  which,  of  many,  contracts  we  will 
pursue.  We  are  profitable.  We  can  grow  our 
capacity  at  a  rate  we  choose. 

in  order  to  ... 

Objective 


f  ^ 

c 

\ 

Cc. 

4.  Do  mature,  correct  things  that  have 

* - 

Dc. 

proven  to  be  very  effective  over  time 

1  ¥ 

t 

2.  Use  the  best  LEAN  tools  and 
techniques,  El  (Employee  Involvement) 

Assumptions 

1 

J 

1.  Toyota  like  methods  have  been  up  to  this  point  the  company  accepted  method 
for  continuous  improvement. 

2.  Toyota  is  considered  to  be  successful  with  their  implementation  of  Lean 
Processes 

3.  The  accepted  site  improvement  methodology  has  been  El 

4.  It  is  easier  to  stay  with  the  current  culture 

5.  El  teams  are  easy  to  measure  and  control 


Train  Lean  Facilitators  on 
TOC 


Create  meaningful  improvement 
metrics  associated  with  Increasing 
Value  and  decreasing  Time 


We  need  to  ... 
Requirement 


Find  other  ways  to  implement  Employee  Involvement 
(remove  the  obstacles  to  being  fast  and  let  the  work 
itself  be  the  reward) 
in  orcier  to...  we  neea  to  , 

Prerequisite 


10th  Annual  National  Defense  Industrial  Association 


CMMI®  Technology  Conference  and  User  Group 

November  16,  2010 
Denver,  Colqjado,  USA 

track  1:  CMMI  anoLProcess  Improvement 
Session  11314,  Chasm  Creek  1:30-2:15  pm 
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S  Bill  Smith,  CEO 

rocess  Consultants  LLC 
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Leading 
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10th  Annual  National  Defense  Industrial  Association 


CMMI®  Technology  Conference  and  User  Group 

November  16,  2010 
Denver,  Colqjado,  USA 

track  1:  CMMI  anoLProcess  Improvement 
Session  11314,  Chasm  Creek  1:30-2:15  pm 


6iil®  m  THE  SOCIAL 


MEDlfcCHALLENGEDI) 


S  Bill  Smith,  CEO 

rocess  Consultants  LLC 
www.  CmrrtFTcai  n  i  ng .  com 


Leading 


J 


Introduction 


j=  > 

PROCESS 
CONSULTANTS  LLC 


■JUlftEllMlM 


n 


An  example  of 
social  media! 


Wikipedia)defines 


PROCESS 

CONSULTANTS 


“media  for  social  interaction, 
using  highly  accessible  and  scalable 
publishing  techniques...” 
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“They  are  re 

inexpensive  and  accessible 
;o  enable  anyone 
(eveji  private  individuals) 
to  publish  or  access  information...” 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conf:  CMMI  in  the  Social  Media... 


Leading 

Edge 

PROCESS 
CO NSU  LTANTS  u 


5 


compared  to 


Introductio 
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PROCESS 
CONSULTANTS  LLC 


That’s  one  definition... 
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www.CmmiTraining.com 
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Here  are  a  few 
other  things 

can  be... 


PROCESS 

CONSULTANTS 
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Introductio  WthM 
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ainmg.com 


PROCESS 
CONSULTANTS  LLCZ 


This  presentation  focuses  on 
how  to  use 

effectively  and  efficiently 
to  enhance  your 
understanding  and  application 

of  the  CMMI® 


■JUlftEllMlM 
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Introductio 

n _ 


We’ll  focus  on... 


Linked  [fl. 


and  provide  a  case  study  on  using  social 
media  for  customer  support. 

The  Leading  Edge  Alumni  Club 
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PROCESS 
CONSULTANTS  LLC 


Social  Media  Applications 

Linkodln 


*•’  '  •  .  »  /  ,  V 

url 


www.linkedin.com 


*£2 


=j|ft^c=L=  I|fe 

w,<*  ^ 

§fKt$ 

**  his ' t 


May  2003 


80  million 


#29 


PROCESS 

CONSULTANTS 


Business  and  professional 
networking 
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Relevance  to  CMMI® 

1  core 


> 


PROCESS 

CONSULTANTS 


,  oevEudpmcnt.  VCRS.' 


ON  1.Z  ▼▼ 


CMMI* 

Second  Edition 

o 

“  Guidelines  for 
:  process  Integration  and 

*  Product  Improvement 


Mary  Beth  Chrissis 
Mike  Konrad 

Sandy  Shrum 


®  Gathering  expert  opinion 

•  interpretational  guidance 

®  Staying  informed 

•  upcoming  events 

®  Filling  a  position 

•  employee,  consultant,  appraiser 


Uh-oh...  you’re  the  SEPG  Lead  and  your 
organization  didn’t  earn  their  desired 
Maturity  Level  rating... 


®  Finding  a 
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How  to  Use 
Linkedln 


PROCESS 
CONSULTANTS  LLC 


Establish  a 
profile 


IP  Smith  Welcome  to  my  Jan.  1$-2l  Intro  to  CWMl-DEV  v  1 .3 cfaaa ,  Selina 
d  John  from  Washington  DC! 

■M  SEl  Inlmduclian  ia  CMMl  DEV  vl.3  January  19.21,  2D11 
— ,  Huston, ...  r!"’™fnln^3  com 

t  j  o  n  aware-winr-ng  speaker  Sul  Sit  ih  for  th.-s  engaging  three- 
day  Introduction  to  the  CMMl  {CWMIdCVl  at  our  alunrlng 
new  facility  near  Washington,  DC. 


at  Leading  Edge  Process  Consultants 


Past 

*  Eiaiinr  Corssulkirri  and  K-F 1  Ai/Hnjfmsd  DM  Ml  IriMrucErr  ;rl 
Eynlisn:;  iind  EMhuim  fim'awliuiri 

lutisgnilisil  Tuohndkigy 

*  Eiaiinr  Corasulhirrl  al  Abrinix;  Tis'hruingy  f 

Ecu  carl  or 

*  E  hi  malign  High  Scheldt 
■  UrevoraiLy  of  PiLirburglh 

*  Carnegie  Mellon  Unlverady 

Recommends  bona 

25  piiscipln  1  ix.i  :  ri  jcnmm uniiiaH  flw 

[jnniriljHTiii 

'J'&j  Eonrat  lions 

Websites 

*  CMMl  Ciasa  Schedule 

*  CMMl  Racks'  (Alumni  Nerewk] 
i  CMMl  Diary  fElog} 

Twitter 

CmmlRiEM 

Public  Profile 

h  11  p h/tutiM,  G  n  kn  1  i  n .  n  cvnfin/tA i  mrl  h  1 1  s  id  i  rgrsi  tjn 

H 


Make 

connections 


Perform 

searches 


Principal  and  Owner  - 
UpStart  Systems,  LLC 

Ahern,  Dennis  24 

Mgr.,  Process  Imp.  & 

Industry  Initiatives  •  Northrop 
Grumman  Corporation 

Allen,  Wade  ns 

Enterprise  Solutions  Architect 

-  Permuta  Technologies,  Inc. 

Allison,  Drew  21 

Principle  Consultant,  Process 
Improvement  -  SSCI 

Alvarez,  Roseann 

Senior  Principle  Systems 
Engineer  -  Orbital  Sciences 
Corporation 

Andersson,  Jim  L. 

Senior  Recruitment  Manager 

-  Wipro  Technologies 

Atkinson,  Shane  iib 

President  •  Atkott,  Inc 


|  My  Groups  (27) 

SE 


Carnog  c  Mcikl 

ActMly:  DdH.iipOff 
tkiw  »  A:1ote  - 


Join 

groups 


UnfvtriKy  of  Pitts b u rgh  A!u mni  Associatto n  orpicxl 


ft! 


Alexander  Kondakov  ■ 
fndgfHmdniil  CwmiPlant  at  CMMl  Krnr'tfafcav 
FktsHiar  |  oeirpjiir  SMwih 

tvn,l,«3  InHiudcr  and  Land  Apemnar  |CMHI  OEV  prtS 
LM  Cana U 1 1. in,  Prulruv,  Cmti'Ml  hnslnKHv m  Land 
In  CiJinrr  ''''  -  t  llvund  ravinr liars  ■  E  ihbVDd  pikipq 
KobJ  Vidtif  ■ 


CMMJ  Bid  Ph9cn>  Irnpramwrfl  Mortar  at  GCV 

Mwntmr  Jd  ,l£MP  Isniri.  Guuulltffl  al  KPI  ft  CMMl  CWurM  ■ 

Ih  {krmrtfi.  +  E  ihamd  crrviodJcni.  *  t  shared  uroona 

Wlguol  Serrano 

Curse™  it  5E0*rTl  (CM  Ml ,  SCAMPI.  FTlL,  PSP,  tSP ,  PSM-l 
.  SL  Louis  MO  USA  and  Urn  Mtmcu  .  www.Monli.cnn> 

□rbaiar  Si  Louis  Anas  |  IrdamMhan  Tnrhraragy  Survlcfl* 

QmciLf  m  atom i  {cum  so.mpj,  itil,  psp,  rap  psmi  , 

AArfltf  PWHWCSPt,  CMMl.  SCAMPI.  PSP/TSP.  P5M  •: 

ir  lYjffltpan  *  2aiuu«d  esmeheni  ■  S  Hhiarad  gmojis 


Agile  CMMl 

PGWflTM  Discussions  Members 


Agile  and  CMMl  levels  4  and  5 

Hi  there, 


Ask 


questions 


I  wonder  what  is  your  opinion  regarding  having  Agile  Methodologies  supporting  CMMl  High 
Maturity  Levels,  namely  levels  4  and  5.  I  read  an  article  about  the  subject  but  it  only  supported 
levels  2  and  3,  it  was  mapped  with  those  levels. 

How  can  Agile  Methods  support  OPP,  QPM,  OID  and  CAR? 

Best  regards, 


Simplified,  and  only  focusing  on 
a  few  key  uses 


ClflMi-CapatiJEiry  Maturity  Model  Integration  [Main  Group] 

AciWily!  Dharfslun*  fl)  .Iota  fi ) 

Oil  m  -  Aruu  - 
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Key  CMMI®-Related 

GrouDS 


Virtually  no  SPAM 


Agile  CMMI 


Discussions  Members  Promotions  Jobs  Search  More. 


^ “  Limited  SPAM 

CMMI-Capability  Maturity  Model  Integration  (Main  Group) 


Members  Promotions  Jobs  Search  More. 


Discussions 


members 


members 


2285 

members 


CMMI  Adopters 

Discussions  Members  Promotions  Jobs  Search  More, 


CMM  -  Integrated  (CMMI,  1500+) 


Group  Discussions  Members  Promotions  Jobs  Search  More. 


Membership  totals  as  of  11/2/2010 
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Specialized  CMMI®-Related 


PROCESS 

CONSULTANTS 


f^rniinQ 


283 

members 


CMMI  for  Services 


fur  ku^Trcvi 


Discussions  Members  Promotions  Jobs  Search  More. 


members 


P&imp-acq  CMMI  for  Acciuisition 


fen  dwuiilllun 


Discussions  Members  Promotions  Jobs  Search  More. 


131 

members 


Oil  Mi 

High  Maturity 


CMMI  High  Maturity 


Discussions  Members  Promotions  Jobs  Search  More... 


262 

members 


myitlfyiflf  Demystifying  CMMI  High  Maturity 

CM Mt,p 

hi  |h  Maturity  Discussions  Members  Promotions  Jobs  Search  More. 


Membership  totals  as  of  11/2/2010 
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Also  of 


PROCESS 

CONSULTANTS 


2048 

members 


National  Defense  Industrial  Association  (NDiA) 

Discussions  Members  Promotions  Jobs  Search  More. 


364 

members 


#-  SEI  Membership 

PTee  Membership 

Discussions  Members  Promotions  Jobs  Search  More.. 


188 

members 


-M 


SEI  Partner  Network 

Farmer  Network 

Discussions  Members  Promotions  Jobs  Search  More. 


Membership  totals  as  of  11/11/2010 
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Example  Use 


[i] 


Lea 

—Edg  I 

PROCESS 
CONSULTANTS  LLCZ 


This  question  about  organizational 
policies  (CMMI®  GP  2.1)... 

CMMi-Capability  Maturity  Model  Integration  (Main  Group) 

Discussions  Members  Promotions  Jobs  Search  More.  . 


How  to  ensure  that  GP2.1  is  satisfied  in  an  organization? 

Hi  All, 

Can  anyone  give  an  example  of  how  the  GP2.1  "Establish  and  maintain  an  organizational 
policy  for  planning  and  performing  the  process."  would  be  incorporated  in  the  organizational 
KiUl  QMS. 

Follow  Rajadurai  E.g.  let  us  take  a  REQM  PA,  if  an  organization  has  defined/established  a  "Guideline  for 
Requirements  Management"  which  needs  to  be  practiced  across  all  the  projects  in  the 
organization.  In  this  scenario  can  we  conclude  that  the  GP2. 1  for  REQM  is  satisfied  with  this 
guideline? 

Thanks  &  Regards 
Rajadurai.A 

1  month  ago 


{b  Like 


P  Comment 


Cl  Follow 


Flag 


More 
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Example  Use 


[2] 


Lea 

—Edg  I 

PROCESS 
CONSULTANTS  LLCZ 


30  comments  *  Jump  to  most  recent  comments 


Follow  Mathi  E 


Mathi  E  Arasu  ■  Hi  Raja. 

Establishing  a  Policy  conveys  only  an  intent  to  perform. 

However,  various  other  GPs  like  2.2,  2.3... to  2. 10  and  also  3.1  and  3.2 
-  when  implemented  will  imply  the  complete  implementation  of  GP  2  A 

So,  with  your  example,  If  you  have  a  Policy  for  Reqiif 
the  implementation  of  this  policy  is  planned,  assigned 
responsibilities  are  defined,  required  train  ina  provided^ 
stakeholder 
reviewed  i 

process  am  (T3  Follow  Lyn 

Managerr 


Hope  this  I 

Cheers, 
Mathi  Ezhil 

1  •month  agT 


o 

A 

Follow  Rajadurai 


Rajadurai  Arunae 
implementing  all  tl 
particular  GP„ 


In  my  case,  weV 
(which  describes 
referred  by  the 
kind  of  documen 
define  a  policy 

Need  to  understi 

1  mon  th  ago 


Follow  David 


David  Walker  -  Y- 
are  no  surprises® 
an  expectation  of 
guideline"  in  your 

1  month  ago 


Lyn  Jones  ■  Rajadurai 

I  guess  you  are  implementing  the  CM  Ml  in  order  to  succeeed  with  an  Appraisal.  Therefore 
my  advise  would  be  in  support  of  David  Walkers  suggestion,  ask  the  Appraiser,  then 
there  are  no  surprises.  If  this  is  not  so  the  case  then  my  advise  would  be  that  if  you  have 
any  kind  of  statement  from  the  management  in  regards  to  how  they  intend  a  specific 
process  or  set  of  processes  to  operate  then  take  that  no  matter  what  its  called  and 
reference  that  as  your  policy  statement. 

My  principle  is  if  it  exists,  use  it,  far  too  many  companies  invent  things  u necessary  in 
order  to  satisfy  the  "requirements"  of  the  CM  Ml.  This  is  extra  work  for  no  added  value  to 
the  organisation  especially  when  something  already  exists  which  will  equally  satisfy  the 
requirement. 

Another  suggestion  that  I  would  recommend  here  is  if  you  are  either  using  existing  or 
creating  from  new  DO  NOT  PLEASE  DO  NOT  make  a  policy  statement  per  Process 
Area,  too  many  companies  do  this  and  it  is  so  obviously  artificial  and  invented  for  the 
purpose,  my  suggestion  would  be  to  possibly  have  a  generic  Engineering  Policy 
statement  which  covers  all  your  engineering  processes,  maybe  one  Project  Management 
Policy  covering  PP,  PMC,  IPM  and  maybe  RSKM.  Taking  this  approach  you  will  end  up 
with  about  4  or  5  policy  statements  which  are  fully  integrated  as  opposed  to  almost  20 
disjointed  policies. 

Hope  this  helps,  if  you  need  further  help  email  me  on 
ly  n .  jones  @  pn  ltd  .org.  u  k 

Thanks 


Lyn 

1  month  ago 


Reply  privately  ■  Flag  as  inappropriate 
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Sample  Questions 


PROCESS 
CO NSU  LTANTS  LL.C 


Set  peer  and  SP  3.1 


From  the  CMMI- 
Capability  Maturity 
Model  Integration 
group 


,cp22  Conauci  r 

^VeSon  within  the  venation  area? 

nosted  2  days  a9° 

— — -  Is  there  any  one  from  here  who  can  lead  the  course  "Official  intro  to  CMMI"  ? 
in  VN?  if  yes,  let's  contact  me,  Thanks 

posted  4  day®  ago 

tendard  4  strategies  for 


a  asked  to  state  the  8W»»  “  rs„. 

lnareCenVnt^CMMiai  answered  this  based 
implementing  the 

posted  t  n 


Why  do  you  need  PP  SP2,3  -  "Plan  for  Data  Management"  when  most  of  the 
things  are  same  in  CM  SP  1.2  -  "Establish  Configuration  Management... 

posted  2  months  ego 

lareW/6  — 


Why  RequiremenUtoiagement  and  Requirement  Deveiopmen 
differnt  PAs  . 

posted  3  months  ago 


how  to  charige  the  mindset  of  the  people  who  hate  process  approach  (other 
than  escalation)  during  the  process  improvement  journey. 

posted  5  months  ago 
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tai  j 

—  Edge 


®  Well-represented 

CMMI®  community 

®  Free  interpretational 
guidance  from 
experts 

®  Specifically  designed 

for  easiness  - 

networking 

®  ^asy  to  use 


®  Too  much  spam  in 
some  groups 

®  Negative  or 

j  competitive  tone  of 
some  discussion 

posts 
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Social  Media  Applications 

Twitter 


Twitter 


About 


twitter.com 

July  2006 

160  million 

#10 

Micro-blogging 


PROCESS 

CONSULTANTS 
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rTWA«*  *NO 


VERSION 


CMMI’ 


CMMI 


Second  Edition 

Guidelines  for 
Process  Integration  and 
Product  Improvement 


Mary  Beth  Chrissts 
Mike  Konrad 
Sandy  Shrum 


If  you  think  the  CMMI®  is  the  best  thing 
since  sliced  bread... 


®  Being  exposed  to  a 
diversity  of  viewpoints 

•  chatter. 


Relevance  to  CMMI® 


PROCESS 
CO NSU  LTANTS  LLC 


®  Gathering  information 

•  useful  hyperlinks 

®  Following  a  conference 

•  live  tweeting 

®  Getting  to  know  new  people 

•  valuable  contacts 
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How  to  Use  Twitter 


PROCESS 
CO NSU  LTANTS  LLC 


You  “follow” 
people 


*nt).  Amateur  Video 
me  Business  Analyst,  if  logger.  iPhone/iPad  Game 
developer/producer :)... 

sdelesie  elena  Deiesie 

rjfl  Test  Manager,  Trainer,  and  Coach  -  Focusing  on 
fM  People.  Learning.  Organizations. 

ISDonline  ISD  Online 


pendulait  °endulalT 

pencJula  it  Pendula  IT  is  an  offshore  company,  based  in  Chennai, 
India,  focused  on  BI/DWH,  BPM  etc. 

.  Dafaso  Danny 

jHLj  Pensioenen,  PHP,fotografie,  photoshop,  robotica, 

motoren,  cradle  to  cradle,  skaten,  istqb,  cmmi,  Almere 


Set  up  a 
profile 


Bill  Smith 

©CmmiRox  Reston,  Virginia,  USA 

CEO  &  CMMI  Entertainer!  (Huh?)  Energizing  trainer  of 
800+  people  in  the  Softu'aiv  Engineering  Institute's 
CMMI.  Loves  rock  music,  dry  humor,  *real*  impiovemenl 


* 

Monitor  the 
action 


I  strongly  recommend  a 
twitter  “client”  (e.g., 
TweetDeck) 


■■  IT  Project  Mana 

Integration  projects" 


People 
“follow” 


you 


cloudjobs  Cloud  Computing  Jobs 

Cloud  Computing  Jobs 


mfalce  Michele  Falce 

SKI  Technical  Event  Coordinator 


777auditfragen  Power  Manager 

Fuhrungsinstrument  Audit 


RogueCFO  Rogue  CFO  Chris  B 

Rogue  CFO,  Management  Consultant.  Helping  growing 
companies  by  bringing  outsourced  CFO  management 
talent  to  your  team. 


Simplified,  and  only  focusing  on  a  few  key  uses 
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Twitter 


How  to  Use  Twitter 


A  search  for  all  tweets 
including  “cmmi” 


A  simple  status 
update 


“Direct 

Message” 

(DM)conversati 


Track  tweets  of 
selected  groups 
of  people 


See  when  you’n 
mentioned  in  a 
tweet 


Monitor  search 
/terms 


Send/receive 
direct  messages 


Search:  cmmi 


Mentions 


Direct  Messages 


SEPG^fOID  Medellin  -  CMMI 
■W^Tcmmi  VT.2 


@CmmiRox  perhaps  u  should 
examine  ur  razor"s  mean-time 
between  failure.  2-sided  T  test? 


Hope  to  see  you  in  Denver  Bill  at 
the  CMMI  Tech.Conf.  Duncan 


Are  executives  aware? , .  .most  u 
p  roj  ect  ma  nag  ers  th  in  k  their  jo  b 
is  to  deliver  the  agreed 
requirements,,,  not  necessarily 
to  deliver  value  4m  ^ 


CMMIA| 
5at  30C 


HETHODfl^il+1 

Thu  Umov  07:23  via  Direct  Message 


RT  @CmmiRox:  Watts  Humphry 
the  "Father  of  software  Quality/' 
pa  ssed  away  tod  ay.  The  SEN  oses 
:  J S  an  icon.  http://bit.Kf/9THEiM 
flcmmi 


TCaglevi  Re:  Podcast.  Yes,  aftei^ 
my  Feb  Las  Vegas  class.  If  we 
emphasize  the  "training"  part.  (U 
could  choose  from  LOTS  of 
people  for  just  CMMI.) 


Raising^he  Voice  of  Quality  -  a 


tabiocabral,  |+| 
ThuI&Qg^ 


CmmiRox,  [+1 

Mon  OS  Nov  OS:  12  via  Direct  Message 


Thu  1 1  NovOYdB  via  Twitter  for  iPhom 


■gCmmiRox  the  class  is  in  for  a 
surprise  then!!  Have  funl 


WOuld  you  like  to  do  an 
interview  on  my  podcast  early 
next  year  or  late  this  year  on 
CMMI /CMMI  Training? 


Nifty  list  of  CMMI  appraisal- 
related  links: 


Is  there  a  back  button  in  IE  for 


TCagley,  1+^^^ 
Mon  OB  Nov  Q2-JT 


♦  CmmiRoK,  |+| 

Thu  1 1  Nov 09:29 via  TweetDeck 


Simplified,  and  only  focusing  on  a  few  key  uses 


Images  taken  from  TweetDeck 


r- 

Re-Tweet”  (RT 

') 

CMMI  y  ^crum  no  se  usan  juntos 
por  falta  de  conocimiento  y 

■ 

creatividad  -  David  Arteaga 
tfSEPG  J^EPGLA 

“’At’  Reply  (@) 

_ 
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-v' 


Twitter 


eading 

Edga 

ROCESS 
ONSULTANT 


Yes,  I 
have  a 
dedicated 
monitor 
for  social 
media... 


Example  Twitter  Client 


-  >^r-^nn  r— ■«!■  . ^ 

****** 


Facebook 


...but 
admitting 
you  have 
a  problem 
is  the  first 
step 
toward 
solving  it! 


Linkedln 
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“Follow”? 


Suggestions  based 

roughly 

•^MMI  relevancy) 

•  number  o^mgwer^ 

•  number  cCtweetsV— 

•  number  of  timeiliste 


ppraiser 


mmi  <ox 


fc  User  Profile 


fc  User  Profile 


fe  User  Profile 


.'H  Bill  Smith 


Jeff  Dalton 

@CMMLApp  raiser 


@CmmiRox 


CEO  &  CMMI  Entertainer!  (Huh?) 
Energizing  trainer  of  BQ0+  peopli 
in  the  Software  Engineering 
Institute's  C MMI. Loves  rock 
music,  dry  humor,  Veal* 
improvement. 


Got  CMMI  q  uesti  o  n  s?  Get 
answers!,  Agile  pioneer  in  the 
CMMI  world,  CM  Ml  Lead 
Appraiser,  Scrum  Master,  and 
CMMI  consultant 
www.broadswo  rd  sol  uti  o  ns.co  m 


I  build  high  performance 
organizations  out  of  companies 
motivated  to  be  lean,  agile,  and 
achieve  world-class  results. 


Friend 


Friend 


9  Ba  Itim  ore.  Or  a  way  o  n  b  iz. 


httpt/ /wwwaq  i  I  ecm  m  Leo  m 
Twitter  page 


9  Reston,  Virginia,  USA 


9  Michigan 


.CmmiTraininaxom/cmmi-instructor.html 


Twitter  page 


Twitter  page 


1426 


Followers  Following 


Tweets^ 


Followers 


@TCagley 


k  User  Profile 


@SEINews 


k  User  Profile 


Thomas  Cagley 

@TCagley 

Process  Improvement 
Consultant,  Blogger,  Podcaster 
and  Family  Guy 


SEI  News 

@SEInew5 


Software 

Engineering 

Institute 

SEI  News 


News  from  the  Carnegie  Mellon 
Software  Engineering  Institute 


Friend 


9  Avon  Lake,  OH 


9  Pittsburgh,  PA,  USA 


Friend 


httpy/www.tcaq  I  ey  .wo  rd  p  ress.co  m 
Twitter  page 


Followers^F  Following 


.Listed 


Followers  Following 
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12 

1973  1 

1449 

Tweets 

Listed 

^Follower^J 

f  Following 

Number  of  CMMI  SCAMPI  A  Appraisals 
increased  sharply,  from  1001  in  2008  to 
1352  in  2009. 

11:56  AM  May  7th  via  TweetDeck 


Bill  Smith 


'09  CMMI  SCAMPI  A  Appraisals:  Asia 
2648,  N  America- 1727,  Europe-709,  S 
America-313,  Africa-63,  Australia-39. 
C'mon  Antarctica! 

1:29  PM  Apr  29th  via  Linkedln 
Ftetvveeted  by  1  person 


©  2010  Twitter 


©2010  Twitter 

■s  API  Business  help  Jobs  Terms  Privacy 

Twit 
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i 

Mgek 

octmpie  i  weeib 
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PROCESS 
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Information 

ac 

er 

1,1 

Bn  C3U5^ H 

Sample  Tweets 


Hyperlinks 


PRocess  Improvement  and  Software 
Engineering  (PRISE)  Search  Engine: 


Discussion:  "Path  to  CMMI  Lead 
Appraiser."  (My  answer  to  a  former 
student's  question.) 


©  2010  Twitter 


10:49  AM  May  8th  via  Tweet  [>eck. 


Bill  Smith 


©  2010  Twitter 


rwitter 


Sample  Tweets 


PROCESS 
CONSULTANTS  , 


Live  from  conferences  or 
other  gatherings 


FLASH!  New  PA  to  be  added  at  CMMI 
ML5  -  Organizational  Performance 
Management  (OPM).  Live  from 
SEPGNA 


2:52  PM  Mar  24th  via  SJmptyTyveet 
Retweeted  by  2  people 


©  2010  Twitter 


:ycie  moaei  *  pee 


♦  quam 


^  evidence  *  operational  scenario 


erlficatio 


>n  eau! 


Agile  interpretation  guidance  will  be 
added  to  CMMI  vi.3.  (Yay!)  Live  from 


2:46  PM  Mar  24th  viaSimp^Tweet 
Retweeted  by  3  people 


3  2010  Twitter 


product  component  *  institutionalization 
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Sample  Tweets 


“Words  of  wisdom” 


Process  improvement  is  NOT 
synonymous  with  making  yourself  more 
"compliant." 


The  CMMI  never  says  you  need  to  use 
the  whole  CMMI.  Prioritize,  addressing 
your  organizational  pain  points  first. 


4  41  AM  May  5th  via  web 


Bill  Smith 


8:  OS  AM  Apr  29th  via  Linked  in 


Bill  Smith 


Sample  Tweets 


Caffeine-laced  lollipops  helped  the 
CMMI  architecture  discussion  go  down 
a  bit  easier  in  today's  Intro  to  CMMI 
class. 

6:55  PM  Apr  Uth  via  linkedln 

Bill  Smith 


©  20 10  Twitter 


cycle  model  *  peer  review 


*  quantiul 
a  c 
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Sample  Tweets 


Humor 


Retiring  from  CMMI.  No  wait  I’m  not. 
Yes  I  am.  Hey  just  kidding...  I'm  not. 
(Sorry,  thought  I  was  Brett  Favre  for  a 
sec.) 

L:4B  PM  Aug  I9th,  2009  via  TweetDeck 


You  know  you're  CMMI  Maturity  Level  l 
when;  Your  most  commonly  applied 
contingency  plan  is  the  phrase  "uh-oh." 
(RSKM)  .U- ml. I.TT 

L  1:2  5  AM  May  6th  via  Tweet  [>ack 
Re  tweeted  by  L  person. 


Bill  Smith 


@  You're  expecting  a  child!? 

Awesome!  And  remember,  CMMI-DEV 
Product  Integration  SP  3.4  covers 
"delivery." :) 

8:4Q  AM  May  I  lth  via  TweetDeck  in  reply  to  spm221 
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Pros  and 
Cons 


®  Useful  hyperlinks  and 
tidbits  of  information 

®  Get  to  know  nfluential 

people 

®  Get  answers  from  your 
followers 

ifoTlow: 


®  Easy  to  “unfollow” 

people  who  waste  your 
time 

®  Follow  conferences 

and  other  events  as 
they  happen 
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®  CMMI®  community 

not  widely 


represented 
®  Need  to  filter  lots  of 


irrelevant  chatter 

®  Need  third-party 
apps  (“twitter 
■ftli^ats’ )  to  be  truly 

useful 

®  Not  intuitive  to 

explain  or  use 


effectively 
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Social  Media  Applications 

race book 


Facebook 

About 


www.facebook.com 


February  2004 


500  million 


#2 


Social  networking 
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Relevance  to  CMMI® 


PROCESS 
CO NSU  LTANTS 


“MT'VE"S,°“’'2  TT 


•  CMMI*. 

Second  Edition 

Guidelines  for 
Process  Integration  and 
Product  Improvement 


Mary  Beth  Chassis 
Mike  Konrad 

Sandy  Shrum 


©  Gathering  information 

•  from  organizations  you  “like” 

©  Solidifying  relationships 

•  personal/business  contacts 


For  now,  limited  CMMI 
relevance.  Included 
here  because  of  its 
widespread  adoption. 
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How  to  Use 
Facebook 


“Friend” 

people 


157  friends  5ee  All 


Hitfel  Henry  Amy 

Glazer  Schneider  Delano 


f 


Paul 

O'Hara 


Simplified,  and  only  focusing  on  a  few  key  uses 


Harold  Jeff  5  mi  Ih 

Taylor 
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Set  up  a 
profile 


* 

Get  status 
updates 


organizations  etc. 


The  Pittsburgh  Steelers 

Sports  /  Athletics 


mn$f 

Leading  Edge  Process  Consultants 
•  Technology  Product  /  Service 


Wired 

Technology  Product  /  Service 


Babicz  Guitars  USA 
Products 
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Add  to  My  Page's  Favorites 
Suggest  to  Friends 
Subscribe  via  SMS 


Remove  from  My  Page's  Favorites 


The  Carnegie  Mellon  Software 
Engineering  Institute  (SEI) 
Membership  program  offers 
opportunities  to  network  with 
leaders  in  software  engineering 
and  security.  Benefits  include  SEI 
course  and  conference  discounts. 


.edu/membership 


Calendar  of 


11  Friends  Like  Thi! 

6  of  11  Friends 


In  this  webini 
All  three  CMJ 
Development 

products,  su( 


October  25  at  L2:52pm  •  Unlike  ■  Comment  •  5hare 


Who  to  “Friend”? 


PROCESS 

CONSULTANTS 


SEI  Membership 

http://www.facebook.com/pages/SEI- 
Membership/1 47935936575 


facebook 


SEI  Membership  Welcome  to  #SEI  Membership:  Kelly  L.  McDaniel  of  Dovel  Technologies  and  Norm  Leclair  of 
the  Software  Engineering  Croup  Hill  Air  Force  Base,  hcu-s  ago 

Wall  Info  Photos  Discussions  Events 


SEI  Membership  Sad 
Medal  of  Technology 
program  at  the  SEI,  c 


SEI  Membership  Don' 
for  the  release?  What 


SO  Membership  Don't  Forget  the  webinar  on  Tuesday:  Are  you  ready 
for  the  release?  What  you  need  to  know  about  CMMI  VI. 3 

Calendar  of  Events  |  Calendar  of  Events 
www.sei.cnnu.edu 

In  this  webinar,  Eileen  Forrester  will  discuss  the  release  of  CMMI  V1.3, 
All  three  CMMI  models,  CMMI  for  Acquisition  (CMMI-ACQ),  CMMI  for 
Development  (CMMI -DEV),  and  CMMI  for  Services  (CMMI -5VC}(  will  be 
released  as  technical  reports  on  November  I.  A  series  of  other 
products,  such  as  training,  5 
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Leading  Edge  Process  Consultants 

http://www.facebook.com/pages/Leading-Edge- 
Process-Consultants/1 20351 941356797 


facebook  i 


Leading  k 

Edgor 

PROCESS 

CONSULTANTS 

Edit  Page 

Promote  with  an  Ad 
Suggest  to  Friends 
Remove  from  My  Page's  Favorites 


Home  Profile  Account  - 


Leading  Edge  Process  Consultants  We're  moving  to  a  larger  locatioi 
Wall  Info  Photos  Discussions  Events  + 


What's  on 

your  mind? 

Attach:  <0] 

a  a 

| 

ilMg  Leading  Edge  Process  Consultants  Found  this  cool  "process 
improvement"  search  engine: 

***  Google  Custom  Search  -  PRISE  -  PRocess  Improvement  and  Software 


lllllif 

Vdgt 


Leading  Edge  Process  Consultants 

Found  this  cool  ’process  improvement"  search  engine: 

Coogle  Custom  Search  -  PRISE  -  PRocess  Improvement  and 

Software  Engineering  (alpha) 

www.google.com 

searches  sites  including,  http://eusescoflsortEiJiTi.org/* 
www.earnedschedule.com  .  www.earnedvaluemanagement.com  f 
http:  / /www. b  sp  i  n.ofg  /  a  rc  he  i  ve  s  L  L ,  http :  /  / www.s ei .cmu.edu /  collabo r . 

iQ]  13  hours  ago  ■  oi  2  •  Like  ■  Comment  ■  Share 


e  moving  to  a  larger 


Get  more  people  to  like 
your  Page  with  Facebook 
Ads!  Preview  below. 


A  certified  Partner  of  the 
Software  Engineering 
institute  (SEI).  Nationally 
recognized  for  award  - 
winning  training  in  the 
CMMI,  the... 

Bill  Smith  likes  this. 

Like 


An  admission:  Unlike  Twitter  and  Linkedln,  I’ve  only  been  on  Facebook  since  early  October  2010.  There  could  indeed  be 
other  disseminators  of  CMMI®-relevant  information  that  I  just  don’t  know  of  yet.  Know  any?  Please  drop  me  a  line. 
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®  Some  useful 

information 


®  Deepen  relationships 
with  friends 

f 

®  ?idel  adopted 
®  to  use 


=  S5"> 

PROCESS 
CONSULTANTS  LLC 


®  CMMI®  community 

not  widely 
m  represented 
(professionally) 

®  Difficult  to  determine 

who  to  “friend”  for 
CMMI  guidance 

®  Must  sift  through 

marketing  updates  to 

get  to  informative 
updates 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


201 0  NDIA  CMMI  Technology  Conf:  CMMI  in  the  Social  Media. . .  46 


PROCESS 
CONSULTANTS  LLC 


Social  Media  Applications 

CMMI  Rocks! 


7/  ftocfe/ 

Warning 

j _ 


PROCESS 
CONSULTANTS  , 


Rt 


CMMI  Rocks!  is  my  network... 

■  ■  ■  that  I  develonnri  *  — — — 

...to  provide  my  CMMI  students  with  a 
base  level  of  support  after  class  has 

ended. 

Ibout  it  "°  Way  1  Can  b®  total|y  objective 

^ctWaPP^-J 


Please  accept  these  slides 

the  spirit 

intended.. 


which  they’re 


in 


.and  it  may 


JN| 


i  X5 


'a: 


n 


s  Consultants  LLC 


2010  NDIA  WlMkiechnology  Conf:  CMMI  in  the  Social  Media 


Rocks! 

About 


cmmirocks.ning.com 

January  2010 

150 


PROCESS 

CONSULTANTS 


The  Leading  Edge  Alumni  Club 


(Are  you  kidding  me?) 


Customer  support  and  (private 
audience)  professional  networking 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conf:  CMMI  in  the  Social  Media.. 


Relevance  to  CMMI® 

1  core 


PROCESS 

CONSULTANTS 


,  oevetOPMENT,  VERS, 


ON  1.2  ▼▼ 


CMMI*. 

Second  Edition 

Guidelines  for 
Process  Integration  and 
.  Product  Improvement 


Mary  Beth  Chrissis 
Mike  Konrad 
Sandy  Shrum 


®  Private  network,  to 
support  students  who 
have  attended  one 
company’s  CMMI® 
classes 
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CMMI  Rocks!  is  a  unique  online  community  of  Leading  Edge  Process  Consultants 
CMMI  class  alumni.  Network  with  your  fellow  students  from  dozens  of  classes, 
attend  special  members-only  events,  and  access  links  to  cool  CMMI  resources. 
Membership  is  limited  to  graduates  of  our  CMMI  courses,  who  must  adhere  to  the 
site  Terms  of  Service.  Non-members  may  browse,  but  cannot  add  content.  (To 
remove  this  restriction,  take  a  class!) 


5igr»  Out 
60  Inbox 
%  Alerts 
f  Friends  -  Invite 
0°  Settings 


Not  Your  Typical  CMMI 
Class... 

Added  by  Bill  Smith 


+  Add  Videos 


Leading  Edge  Links 

■  CMMI  Training  Schedule 

■  CMMI  Diary 

■  Bill  on  Linkedln 

■  Bill  on  Facebook 

■  Bill  on  YouTube 

■  CmmiRox  on  Twitter 
CMMI  Version  1.3  Links 

■  CMMI  for  Development 

■  CMMI  for  Services 

■  CMMI  for  Acquisition 
An  nouncements 

■  Watts  Humphrey,  1927- 
20  LO 

■  Smith  Receives  SEI's 
2010  Outstanding 
Representative  Award 

■  Process  Maturity  Profile, 
March  2010 


Watts  Humphrey,  1927-2010 

For  more  on  the  life  of  this  remarkable  man,  visit  www, sei.cmu.edu/watts 


+  Add  Photos 


2010  NDIACMMI  Conference  2  Replies 

Started  by  Bill  Smith  in  CMMI  and  Process  Improvement  Last  reply  by  Bill 
Smith  Nov  5. 


Find  lead  auditors  6  Replies 

Started  by  Fuji  Nguyen  in  CMMI  Appraisals  .  Last  reply  by  Fuji  Nguyen  Nov  4. 

CONTEST:  Name  My  New  CMMI  Course!  26  Replies 

Startej^j^il^mith  in  CMMI  Training .  Last  reply  bv  Bill  Smith  Nov  2. 

www.CmmiTraining.com 
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The  Leading  Edge  Alumni  Club 


VIDEOS 


Not  Your  Typical 
CMMI  Class,, 


QUICK  UHKS 


PHOTOS 


PROCESS 
CONSULTANTS  LL-C 


Main  Page  - 


View  All 


LATEST  ACTIVITY 


■  Joyce  Danaher  is  now  a 
member  of  CM  Mil  Rocks! 
Welcome  Them! 

arr  Monday 

%  Bill  5mith  added  3  photos 


C  MM1  DIARY 


The  Top  0  New  Concepts  in  CMMI  vl.3  (Part  3) 


B 


November  2010 


S  M  T  W  T  F  S 

12  3  4  5  6 

7  3  9  10  11  12  13 

14  15  16  17  18  19  20 

21  22  23  24  25  26  27 

26  29  30 


art  Sunday 


In  my  past  two  entries  [Oct  22  and  Nov  1),  I  introduced  the  first  seven  of  my  Top 
8  New  Concepts  in  CMMt  vl.3  —  other  than  the  "front  page  news"  of  high 
maturity  clarification  and  harmonizing  the  models. 


Last  Month  Next  Month 

Alumni  Happy  Hour 

December  9,  2010  from  5pm 
to  7:30pm  -  Bill's  Place 

+  Add  an  Event  View  All 


5teve  Ford  is  now  a  member 
of  CMMI  Rocks! 

4  Welcome  Them! 

November  £ 

Bill  Smith  replied  to  Bill 
Smith's  discussion  '2010 
NDIACMMI  Conference' 

Nqvember  5 


0 


CMMIROX  ON  TWITTER 


CmmiRox.  RT  @TCagley:  I  have 
heard  that  Allan  j,  Albrecht  the 
creator  of  £FunctionPoint  has  died, 

CmmiRox:  Nifty  list  of  CMMI 
appraisal- related  links: 
http://bit.ly/bxHPoX 

CmmiRox.  Welcome  to  my  Jan.  19- 
21  Intro  to  CMMI-DEV  vl.3  class, 

Co  I  i  n  3  zi«iH  ffftm  Ubc  hinntrin 


Here's  my  Number  1, 

1.  Agile  Interpretive  Guidance 

Say  the  word  "Agile,"  Great!  Now  say 
"CMMI,"  Super!  Quick  question:  which 

I  sounds  sleeker.,,  smoother,,,  sexier? 

Even  the  authors  of  the  CMMI  would 
have  to  admit  they  lose  that  battle 
every  single  time! 

Now,  I  certainly  can't  claim  to  be  an 
agile  expert,  based  on  the  fact  that 
once-upon-a-time  I  read  a  book  and 
then  I  kinda-sorta  applied  it,  [Along 
with  a  dose  of  the  CMMI  for  Services, 
though,  it  did  help  to  turn  my  little 
company  around.)  But  I'd  be  a  fool  — 
and  so  would  you  --  to  ignore  the 
fact  that  agile  methods  such  as 
5crum  have  taken  the  development 

world  by  storm  in  the  past  decade.  Yet,  agile  developers  often  don't  see  how 
CMMI  can  help  them,  Reasonable  people  have  said  it  can.  and  publications  like 


I'm  a  member  of 

CMMI  Rocks! 


Get  Badge 
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How  to  Use  CMMI 
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CONSULTANTS  LLC 


Attend  a  class 


Create  a  profile 


SpKial  mil  &  -fl 

Ptmjwr  111 

Fnflln<*nnj.  5 riv  r*  EtuMinmi 


Ask  questions 


Gap  Analysis 

ftw«d  ev  Sifin  fl'esipha1  dp  August  4,  20 JO  u  1:42« 
O  id  Mess apt  fl  V«ew  Discussions 


p  CMMI  and  Pr»c(»  Improvement  (Changeli 


Access  resources 


CMMI  Links 

Math  d.t  Jt, i.*it  vMdee  seteert  wty  the  fmess  ettifte  heap' 

related  llhki  An  the  web.  {Of  etniffe,  )  iiltrflV  ytm  j beady  kerns'  hn 
prAgeet y  thetk  biAk Over  Hie  i text  few  weeks  At  l  develnp  pages  A 


-  cutting  Started.  You've  taken  my  Intrn  to  CMMl  d-s-t-t.  f 

-  Appraisals,  Detern: inns  your  cjrren  state.  (Advice:  dp 
*  Training,  lx  doesn't  have  to  stop  with  the  intro  coarse. 

■  networking.  Where  (tie  car  you  meet  hke -minded  peo 

■  Books.  The  book>  I'ee  msnli'Dned  --  and  sometime!  evi 

V  Edit  K  Del  etc  thi*  Nfl* 


\ 


i  wu  wand'erinp  If  anyone  hat  or  knows  of  any  type  of  tools  [spread*  seats 
documents.  «t)  that  are  avalfable  to  help  with  performing  a  gap  analysis  fi 
Dev? 


Attend  events 


Friday,  August  13,  2010 


Alumni  Coffee 

August  13,  2010  from  7:30am  to  9:30am  -  Greenberry’s 
Coffee  &  Tea  Company,  Reston,  VA 
Stop  by  Creenberry’s  Coffee  and  Tea  Company  in  Reston 
if  you'd  like  to  talk  CMMI  or  just  socialize.  CMMI  Rocks! 
members  only,  please.  Limited  to  the  first  four  people 
who  RSVP. 

Organized  by  Bill  Smith  |  Type:  meeting 
You  are  attending. 


ti  bJ/idpiAking  the  mbit  use f of  CMMI  - 
J  get  here.)  Ai  ytto  fdn  see,  th/s  ii  A  whfk  j'ri 
tfdit/bbil  tap /ft. 

Ypar  friendly  neighb&b&Xi  CMMI  iAWuft/Af, 
fkti  £mih 


jtsloft  groups.  SPINS,  etc. 
n.  Plus  p  few  more. 


Receive  broadcast 
messages 


Simplified,  and  only  focusing  on  a  few  key  uses 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conf:  CMMI  in  the  Social  Media... 


53 


CMMI  Rocksj 

Events 


“8  EVENTS  Q 

All  Events 

My  Events 

+  Add  an  Event 

Alumni  Coffee 

i  l 

Added  by  BiFI  Smith 
ft  View  Events 

EVENT  DETAILS 


ADMEN  OPTIONS 


Time:  August  13,  2010  from  7.30am  to  9:30am 


★  Feature 


Location:  Creenberry's  Coffee  &  Tea  Company,  Reston, 
VA 

Street:  11790  A  Baron  Cameron  Avenue 
City /Town:  Reston,  VA  20190 


\  Edit  Event 
AS  Send  Message  to 
Guests 


Website  or  Map:  http://maps.googfe.com/maps7f  . 

Phone:  571-232-0146 

Event  Type:  meeting 

Organized  By:  Bill  Smith 

Latest  Activity:  Aug  8 


X  Delete  Event 


YOUR  R5VP  { 


You  are  attending. 


4r  Export  to  Outlook  or  iCal  (Jcs) 


Change  RSVP 


=  SB-> 

PROCESS 

CONSULTANTS 
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Resource 


> 


PROCESS 
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All  Page* 

Glass 


M V 

Qovjn\oa^s 

.  cMMt  Crossword. 

silences. 

Cross1 


All  Pages 


My  Pages 


Appraisals 


+  Add  a  Page  i 


freely 


process^ 


^embers.  Or,  use  ' 


it-pa\nt 


mini-P(es' 


Wanted  frornl 

mentation,  adapts  ■ 


The  S£/‘s  oifjW/  a  .  +  Add 

though,  would  f  k  Pprais^(s  Homepage  vn 

■p^ziz^ 

*  A"  afternative  cmm,  Q'  Shai9ht  from  the  hn,  ,  ComrrK"1 

irreverent  and  candid^  h*'*  FAQ’  Pan  °f  »he  kind  ™  *  "1°Utk  Cood  stuff. 

Di*ta  bases.  Web  ar  ’  Whyr  1  it!  a“SOrta  fi™°us  "brutalfv  h 

TsssSr.'— .0 

Recaps that ""  9  I  or  peoPle  with  too  “""f  °F  "Wired  practfceV  „  MP‘  Up9^e  Team  n, 

1  ’  «*m#rdd*  fcr  r  t,me  °"  «*»r  hands.  (You  a"d  vafia«°n  9**k's  bib‘* 

>-.Ns . 

eaSI"  to  —  ■  an  MOD.  ^  Uader 


«  UYlwi« 

Miscellaneous  fiUcd 

.  FERpAf°r,nn‘ 

;  this  Page 


orve  out  in 


class 


.Edit  XWW" 
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CMMI  Rocks! 


Sample  Questions 


PROCESS 

CONSULTANTS 


CMMI  and/or  w  been 

Cr^r.“°o"r«earcn  ..«OOHOO.  Our  CEO  a, 

CMM|Ve| 


es 

nCMMI  Version  1.2  vs.  1.3  Training 
Considerations 

11 J  fvhpjr  74-7fi_  7010.  Infmdurrini 


-  ,SO  Auditor  Que*Jj®"„e5tion  «.  "B-i 

S-JSIW “  **•*****.  °" 


ober  24-26,  2010,  Introduction  to  CMMI 
my  final  public  vl.2  class  ever.  Why  is  this 


Ml  Training 


Qd-^lWT  ■— 

,M  apprafsa»d  ^  fer  SCAMPI  «*• A 

wSm  for^SS1! iS  R0M  “  *>  Oass  A.  B 

^  F« 

-/  p  Process  tailoring  nu^whaT 

S  KSIS^WSSSJS- 

“  ■ is..  * 


1  A,  B,  or  C 


55  improvement 

Organizational  Training  Waivers 

What  standards  are  used:  to  waive  a  student  fro 


What  standards  are  used  to  waive  a  student  from  required 
training?  I  am  writing  the  organizational  training  plan  and  in  it 
I  need  to  sped... 

Started  by  Ted  H.  Exstein  in  CMMI  and  Process  Improvement 
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WM1M&EL 


Example  Use 

in _ 


Lea  ik 

—Edg  r 

PROCESS 
CONSULTANTS  LLCZ 


This  question  about  CMMI-SVC  vs. 
ITIL for  an  IT  Helpdesk... 


All  Discussions  My  Discussions 


+  Add  a  Discussion 


CMMI-SVC  or  ITIL  for  IT  helpdesk 

Posted  by  Fuji  Nguyen  on  May  13,  2010  at  5:52pm  in  CMMI  for  Services  (Change) 
Send  Message  ft  View  Discussions 


I  went  to  Bill  CMMI  SVC  class  in  mid  of  April  2010.  As  usual,  instructor  Bill  was  great.  I  learned  a  lot  from  the 
class. 


ADMIN  OPTIONS 


★  Feature 

While  listening  to  Biirs  preaching  of  CMMI-SVC,  I  noticed  many  similarities  between  CMMI-SVC  and  ITIL.  CMMI 

is  broad  while  ITIL  is  specific  for  IT.  If  anyone  has  experiences  with  implementing  either  CMMI-SVC  or  ITIL  in  IT  Edit  Discu  ssion 


environment,  please  share.  I  am  particularly  interested  in  IT  helpdesk  area. 
It's  time  to  practice  what  we  learned  from  Bill. 


*ij  Close  Discussion 
Edit  Your  Tags 
X  Delete  Discussion 
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CMMI  Rocks! 


Example  Use 


PROCESS 
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Replies  to  This  Discussion 

oo  Reply  by  Lita  Fulton  on  May  18,  2010  at  7:20am 


^  Send  Message 

Hi  Fuji,  I  have  c 
subset  of  HI L  t 
significant  in  st 
is  the  only  one 
Function,  CMMI 
of  them  add  res 
Prevention  CHRP] 
the  help  desk. 

As  you  indicate 
CMMI -SVC  mot 
Delivery  (SD)  ar 
part  of  your  sei 
delivery  (your  5 
system  (require 

\  Edit  Comme 
►  Reply  to  This 


...  led  to  several  extensive  replies, 
and  an  eventual  “report”  back  from 
the  original  questioner 


oo  Reply  by  Cathy  Beykzadeh  on  May  18,  2010  at  6:51pm 
£3  Send  Message 
Fuji, 


CMMI  for  Services  is  a  perfect  methodolc 
Here's  a  high-level  approach  for  applying 
of  "helpdesk"  as  a  Service,  Accordinly  to 
by  facilitating  outcomes  customers  want 
risks, 

Project  Planning  (PP>:  Project  Plan  for  est. 
support,  including  people,  process,  techr 
Project  Monitroirtg  and  Control  (PMQ:  Pic 
service  establishment 
Integrated  Project  Management  (IPM):  Int 
initiatives  on  the  helpdesk,  for  example,, 
new  locations;  or  modification  to  current 
Requirements  Management  (REQM):  Man; 
managing  new  and/or  changed  requirem 
Risk  Management  (RSKM):  Managing  youi 
any  update  to  current  functionality  or  est 
Capacity  and  Availability  Management  (C 
helpdesk  establishment,  for  example,  pis 
support  the  current  phone  line  that  the  li 
People,  process,  technology  and  product: 
Organizational  Process  Definition  (OPD): 
process  assets  and  work  environment  st; 
Organizational  Process  Focus  COPT):  Plan, 
improvements  on  the  helpdesk  based  on  a 
weaknesses  of  the  organization's  processes 


oe  Reply  by  Fuji  Nguyen  on  September  2,  2010  at  10:32am 
©  Send  Message 

Thanks  to  Bill,  Cathy,  Lita,  etc.  for  your  guidance. 


Here  is  my  update  on  making  help  desk  more  Service  oriented 

1.  The  term  Service  Desk  was  used  to  describe  the  functional  group  (instead  Help  Desk) 
[map  to  Organizational  Process  Focus  (0PF)] 

2.  Leveraging  free  open  source  DotNetNuke  (www.dotnetnuke.com)  and  aDefHelpDesk 
(http://adefhelpdesk.com)  to  automate  tracking  of  support  tickets. 

[map  to  Service  System  Development  (SSD)  and  Service  System  Transition  (SST}] 

3.  Getting  three  IT  staffs  ITILv3  certified 
[map  to  Organizational  Training  (OT)] 

4.  Updating  existing  help  desk  procedure  to  match  tTIL  v3  Service  Desk  function 
[map  to  Incident  Resolution  and  Prevention  (IRP)] 

5.  Providing  training  to  users,  including  a  brochure  of  available  services. 

[map  to  Service  Delivery  (SD)] 

Attached  please  find  the  process  diagram.  1  hope  the  information  will  be  helpful  to  those  on 
similar  journey. 

\  Edit  Comment 
Attachments: 

g  service_deskjprocess.pdf,  88  KB  X  Delete 
►  Reply  to  This 

thorough  understanding  of  the  current  strengths  and 
and  process  assets 
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®  Logical  extension  of 

the  classroom 

experience 

®  CMMI®-immersive 

environment 

•  designed  for  CM  Ml® 
knowledge 
dissemination 

®  All  members 

gtersonally  know  (and 
presumably  trust)  the 
site  owner 

®  Spam-free 
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Summary 


j=  > 

PROCESS 
CONSULTANTS  LLC 


^ '  '/eading 


®  Common  social  media  sites  include 
Linkedln,  Twitter,  and  Facebook 

®  Social  media  can  used  successfully  for 
customer  support  (CMMI  Rocks!) 

®  Each  of  these  has  its  own  unique 
strengths  and  weaknesses 

®  They  all  may  be  used  to  enhance  your 
understanding  and  application  of  the 
CMMI®. 


My  other  Presentations  This 


Edge 


Coming 
up  next, 
in  this 
room! 


pcEs^rKvou^, 

[  *£32222; 

■ 

Leading  Edge  Pr0ceSs  <?'"  CEO 
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> 


PROCESS 
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Bill,  why  didn’t  you  mention  blogs? 

Or  YouTube? 

Or  other  social  media  applications? 


Then  what  would 
I  talk  about  next  year? 
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Questions 


Intro  to  CMMI  vl  .3 
Hard  Rock  Hotel 
Las  Vegas! 

Feb  8-10,  2011 


Tntro  to  CMMI-DEV  vl.3  ^ 

-  Jan  19-21, 2011,  Reston,  VA 

^eb  8-10,  ,  N \lJ 


Website: 

Blog: 

Alumni  Club 

Twitter: 

YouTube: 

Linkedln: 

Facebook: 


www.  .com 

www.  .com  Bill  Smith 

.ning.com  bill@cmmitraining.com 

twitter.com/ 

www.youtube.com/ 

www.linkedin.com/in/ 

www.facebook.com/pages/ 


120351941356797 


2010  NDIA  CMMI  Technology  Col 
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g.com 


Leading 


CMMI®  Technology  Conference  and  User  Group 

November  16, 2010  f:  \ 

Denver,  Colorado,  USA 

Track  1:  CMMI  and  Process  Improvement 
Session  11315,  Ch^m  Creek  2:15-3:00  pm 

WE1AT  DOESN’T  KILL  Y@P  HAKES  Y@U 
^  STRONGER: 


LEARNED 

_ _ _ _ _  _ _ _  _ 


WE^SI©M  1.® A  (LARGE  FORMAT)  -  10/25/2010 

Bill  Smith,  CEO 

Leading  Edge  Process  Consultants  LLC 

www.CmmiTraining.com 


10th  Annual  National  Defense  Indqstrial  Association 
CMMI®  Technology  Conference  and  User  Group 

November  16, 2010  r  j  " 

I  V  \  *  f 

Denver,  Colorado,  USA 

Track  1:  CMMI  and  Process  Improvement 
Session  11315,  Ch^m  Creek  2:15-3:00  pm 

MEAT  DOESN’T  KILL  ¥©0  HAKES  YOU 
^  STRONGER: 


MY  PROCESS 


Leading. 

_  Film 

PROCESS 
CONSULTANTS 


LEADED 

WE^SI©M  1 JA  (LARGE  FORMAT)  -  iMIi 

Bill  Smith,  CEO 

Leading  Edge  Process  Consultants  LLC 

www.CmmiTraining.com 
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No,  that  wasn’t  me. 
I  have  tattoos... 


...that  I’ll  admit  to. 


WWW.CmmiTraining.com  2010  NDIACMMI  Technology  Conference:  What  Doesn’t  Kill  You...  3 


Introduction 
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Introductio 

n 


I’ve  been 
involved  in 
improvement 
initiatives 
dating  all  th 
way  back 


\ 


...when  I 
used  a 


com 


2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You... 


One 

theme 

has 

emerged... 


PROCESS 

CONSULTANTS 
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Introductio 

n 


a 


Out  of  life’s  school  of  war: 


What  does  not  destroy  me 


I 


-Jr  / 


makes  me  stronger. 


n 


■ 


iWl 


Tl 
Jf  * 


i 


& 


WWW, 


.CmmiTraining.com  DIjWjMMI  Technology 


Friedrich  Nietzsche,  1889. 
Often  paraphrased  as  “What 
doesn’t  kill  you  makes  you 

stronger.” 

Conference:  What  Doesn’t  Kill  You... 
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Or 

to  put  a 
more  modern, 
less  depressing 
spin  on  it... 
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“Next  time  you  feel  like 
It's  just  one  of  those  days 
When  you  just  can't  seem  to  win 


Don't  stay  down ,  try  again ,  yeah 


Everybody  makes  mistakes , 
Everybody  has  those  days.. 


Hannah  Montana 
Nobody’s  Perfect” 

2007 


“Hannah  Montana  aka  Miley  Cyrus  on  the  stage  of  Hannah  Montana  Tour,” 
Author:  Mike  Schmid  from  Hollywood  CA.  Photo  in  public  domain,  used  here 
under  the  following  license:  http://creativecommons.0rg/licenses/by-sa/2.O/ 


■MPj^fcchnology  Conference:  What  Doesn’t  Kill  You 
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n 


So  that  you  can  avoid 
some  of  the  mistakes 
I’ve  seen 

and  sometimes  even 


► 
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I  submit  to  you  my... 
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Introductio 

n 


Process 
Improvement 
Lessons  Learned 
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Lesson  10 
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Background 

•  Organization:  ML  1 ,  new  to  measurement  except  for  the  basics  (e.g.,  cost) 

•  Me:  Newly  appointed  PI  consultant,  in  my  first  SEPG  meeting  w/  them... 


•  Six  months 
planning  /  trying 
to  collect  the 
measures 

•  All  but  2  are 


SEPG  Everyone  Vote  called, 

member  agrees,  measures 

proposes  8  except  one  “adopted” 

new  guy  (a  PM) 

measures...  saying 

for  REQM!  “no  way” 


</> 


eventually 

abandoned 

among  projects, 
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Naysayers; 


Don  t  Ignorg^th 

They  May  Be  Right 


See  also:  “Conflict 
jump-starts  the 
creative  process,” 

Ikujiro  Nonaka 
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Lesson 


10 


Lesson  10 


.  irp  %\  V  ' '  1  > 

■ '  So  ■  ^ 


Leading 

_z Edge 
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Don’t 


See  also:  “Conflict 
jump-starts  the 
creative  process,” 

Ikujiro  Nonaka 


Notes 

That  Project  manager 
was  the  voice  of  reality! 
Q  Value  alternative 
perspectives 

□  Don’t  “steamroll”; 
strive  for  consensus 

□  Satisfy  the  WIFM  for 
key  stakeholders 
(  What’s  In  it  For  Me”) 


I 


They  lOl 


Naysayers; 
y  Be  Right 
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Lesson  9 
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Lesson  9 


Backgroun 
d 

•  Me:  Tech  lead  of  an  8-person  software  development  team 

•  Team:  Recently  started  code  reviews,  everyone’s  OK  with  them  except... 

•  Three  weeks 
later,  he 

on  his  own 
(presumably 


One  guy  He  sits  Immediately  taking  a  job  in 

takes  them  quietly  afterwards,  an  organization 

personally,  through  a  he  repeatedly  ^  where  they  don’t 

always  review  of  his  pounds  a  ^  do  code 

pushing  back  code...  a  refrigerator  reviews!) 

until  1  day...  good  sign  (?)  against  the  ^ 


wall  in 


frustration! 
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sson  9 


slotes  „ 

Respite  our  best  efforts, 

we  lost  a  valuable 
employee;  maybe  we 

were  all  better  off 
□  Remember  the  W1FM 

(Lesson  10),  be 
empathetic,  and  use 
svery  ounce  of  your 
people  skills 
□  But...  don’t  be  too 
hard  on  yourself  when 
someone  just  won  t  get 
with  the  program” 


Leading 

Edge 


Lesson  8 
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Background 

•  Organization:  Assorted  ones,  with  varying  degrees  of  guilt 

•  Me:  A  (youthful)  software  developer 


I’m  told  we 
have 

processes, 
and  I’m 
handed  some 
impressive 
binders 


I  put  the 
binders  on 
my  bookshelf, 
intending  to 
look  at  them 
“when  I  have 
time” 


I  keep  doing 
what  I  was 
doing,  the 
way  I’ve 
always  done 
it 


•  I’m  in 

the  processes 

•  Nobody  seems  to 

if  I  use  them 

•  “Has  anyone 
actually 
these?” 

•  My  bookshelf 
under  the  weight  of 
the  binders 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You...  22 


Lesson  8 
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“And  Then 
a  Miracle 
Happens”.. 
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See  also:  “Train,  train, 
train,”  Watts  Humphrey 


...Is  Not  a 
Deployment  Plan 


f 
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Lesson  8 


Notes 

□  Pilot! 

□  Train! 

□  Enforce!  (Even  better. 

care!)  . 

□  Consider  projects 
existing  commitments 

□  Honor  “legacy 
projects”:  Is  business 
value  enough  to  justify 
change? 

“And  Then 
a  Miracle 
Happens”... 


Copyright  2010 
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...Is  Not  a 
Deployment  Plan 

[  \  r 
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Lesson  7 
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Background 

•  Organization:  (The  less  said,  the  better) 

•  Me:  (I  plead  “the  Fifth”) 


Organization 
believes  they 
must  be 
“compliant”  - 
now!  -  to 
compete 


They  develop 

&  deploy 

process 

descriptions 

as  fast  as 

humanly 

possible 


Eventually, 
leading  up  to 
an  appraisal 


•  Successful  ML  3 
Level  rating! 

•  Several  months 
later,  I  hear  from  a 
POC  that  they’re 

anymore 

•  Back  to 

(Until  the  next 
appraisal?) 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You...  26 


Lesson  7 
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Process 


Improvement 
and  Compliance 

Are  Not 
Synonymous 


See  also:  “Cynicism  and  dysfunction  both 
begin  when  managers  start  saying  one 
thing  and  thinking  another,”  Roger  Martin 
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■-r^vcment  rasf  car) 

eQual  doing  it  wrong 
Q  Avoid  “crappy” 

CMMI®-compliant 

processes 

□  Survival  of  the  fittes 
If  your  processes  don’t 
make  things  better, 
cheaper,  or  faster  thi 
won’t  (&  shouldn’t)  last 


Process 
Improvement 
and  Compliance 

Are  Not 
Synonymous 


See  also:  “Cynicism  and  dysfunction  both 
begin  when  managers  start  saying  one 
thing  and  thinking  another,”  Roger  Martin 
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Events 


Lesson  6 


PROCESS 
CO NSU  LTANTS  LL.C 


IH 


Background 

•  Organization:  ML1,  striving  for  (and  will  eventually  attain) 

•  Me:  Their  process  improvement  consultant 


One  PM  has 
team 
meetings 
...every 
. .  .single 
. .  .day! 


I  suspect 
overkill,  and 
I’m  secretly 
glad  I  don’t 
work  for  that 

guy 


Gradually,  I 
learn  more... 


ML  3 


•  His  project  is  the 
only  1  in  the  org 
to  reliably  deliver 
software  on-time 

•  His  customer  & 
team  love  him 

•  He’s  using 
Scrum,  an  agile 
method 

•  Overkill?  No. 
Potential  best 
practice?  Yes! 
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Lesson 


Leading 

Edge 

PROCESS 
CONSULTANTS  LL.CZ 

Keep  an 
Open 
Mind 

lAliuq 


See  also:  “Keep  an  open 
mind  and  a  level  head,” 

Watts  Humphrey 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You...  31 


Lesson  6 


Leading 

Edge 

PROCESS 
CONSULTANTS  LL.CZ 

Keep  an 
Open 
Mind 

lAliuq 


See  also:  “Keep  an  open 
mind  and  a  level  head,” 

Watts  Humphrey 
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Events 


PROCESS 
CONSULTANTS  LLCZ 


► 


Background 

'  Team:  Three-person  IV&V  team,  has  been  operating  for  about  six  weeks 
1  Me:  Their  new  team  lead 


Team 

member  has 
been  working 
full-time  on 
an  “IV&V 
database” 


But  he  hasn’t 
actually  been 
doing  IV&V,  & 
the  database 
is  constantly 
being 

redesigned! 


I  pull  the  plug 
on  the 
database 
development 


•  We  start 

focusing  on  our 

job  -  to  do  IV&V 
-  instead  of 
building  a 

software 

application 

•  We  begin  using 
a  spreadsheet, 
which  suits  us 
just  fine  for  the 

next  year 
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See  also:  “Improvements 
should  be  made  in  small, 


tested  steps,”  Watts  Humphrey 
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Notes 

A  database  may  nav 
been  fine ...  someday . 
□  Test  improvements 


incrementally,  just  like 
software 

□  innovation  rocks,  t 


See  also:  “Improvements 
should  be  made  in  small, 


tested  steps,”  Watts  Humphrey 


it  intelligently 
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Background 

Organization:  Small  consulting  company,  simply  interested  in  getting  better 
Me:  Quality  Manager,  tasked  to  begin  a  PI  effort  with  little  funding 


I  form  a 
Process 
Group 


We  spend 
weeks 
developing 
our  “process 
for  developing 
a  process” 


It’s  great! 


$  pressures 
(“billability”) 

the  effort 

Created  assets 
that  are  super  in 
theory,  but 

the 

business 
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sson  4 


Leading 

Edge 


COIJIBU 


See  also:  “You  have  choices  in  your  improvement 
approach,”  Suzanne  Garcia,  Richard  Turner 


.  —  -  .  '  ‘ 
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One-S  ze-F  ts-A 


Improvement 


Approach 


%  V. 
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Lesson  4 

*  I.  I 

II 


Leading 

mEdge 


i 

Avoid  a 


Notes 

Twas  doing  PI  the  way 
I’d  seen  it  done  in  large 

corporations 

□  Big  company 

approaches  may  not 

work  in  small  companies 

□  |DEALSM  may  not 
always  be  ideal. 
Consider  other 
approaches  (e-9-  ^ 

Decision-Based  Life 
Cycle  for  Improvement, 
CMMf®  Survival  Guide) 


See  also:  “You  have  choices  in  your  improvement 
approach,”  Suzanne  Garcia,  Richard  Turner 


-  * 
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Approach 
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Background 


See  also:  Lesson  2 


Organization:  Recently  formed  80-person  software  development  organization 
Me:  Newly  appointed  Quality  Assurance  Manager 


My  QA  team 
plans  and 
begins  a 
series  of 
audits 


ljiti  se 
jteretaj 


No  projects  We  try  to  find 

are  following  out  “Why?” 

the 

organizational 

standard 

process! 


Developers  are 
not  undisciplined 
or  lazy  or  in 
need  of  training 

The 

organization’s 

processes 

are(verifiably) 
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Lesson  3 
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Lesson  3 


Leading 

Edge 


Notes 


Q  Don  t  assume  your 
processes  are  being 
ignored  because  people 
are  lazy,  untrained,  etc. 

□  People  may  simply  be 
trying  to  do  a  good  job, 
by  avoiding  your  bad 
processes! 

□  Use  elements  of  CAR, 
even  at  low  maturity 


See  also:  To  identify  actions  that  address 
defect  or  problem,  you  need  to  understand 
its  root  causes,”  CMMI-DEV  vl  .2  (CAR) 


www.CmmiTraining.com 
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Background 


See  also:  Lesson  3 


Organization:  Recently  formed  80-person  software  development  organization 
Me:  Newly  appointed  Quality  Assurance  Manager 


Going  back  to  They’c 

Lesson  3:  f 

Why  did  tthe  teamt 

organ'23  ion  |abore 

develop  bad  weeks 

processes? 


They’d  The  team 

formed  a  tiger  included 

team  that  nobody  that 

labored  for  6  would  be 

weeks  using  the 

processes! 


You  already  know  that: 

The  processes 
were 

A  process 

effort 

began... 

...but  after  several 
months,  the 
organization 

due  to 
poor  performance 
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Lesson  2 


See  also:  “Ultimately, 
everyone  must  be 
involved,”  Watts 
Humphrey 


*~>i  >•,  t  — 
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Lesson  2 


= 


See  also:  “Ultimately, 
everyone  must  be 
involved,”  Watts 
Humphrey 


■--->  ,  fy4  /  ""  " 
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Notes 
Now  I  cringe  in  fear 
when  I  hear  “tiger  team 

□  Avoid  stacking  your 
EPG  with  “ivory  tower” 
experts 

□  Involve  people  who 
will  be  responsible  for 
executing  the  processes 
E)  Include  a  diversity  of 
perspectives;  you’ll  end 
up  with  better  stuff! 


f'l 
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PROCESS 
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Background 

Organization:  Small  company,  nearly  ML2,  moving  towards  ML  3 
Me:  Their  PI  consultant 


PI  effort  way 

behind 

schedule 


We  draw  a 

fishbone 

diagram, 

identify  root 

causes, 

implement 

solutions 


Most 
solutions 
involve 
treating  PI 
like  a  real 
project 


We  rigorously 
estimate  process 
development  time 

We  develop  a 
schedule  down  to 
the  short-duration 
task  level 

We  track  actual 
progress  in  detail 

Successful  ML3 
appraisal  -  on 
time! 
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Lesson 


1 


I 


See  also:  “Unplanned  process 
improvement  is  wishful  thinking,’’ 

Watts  Humphrey 

L  I 
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Lesson  1 


Notes 

□  Make  PI  your  mos\ 
important  project,  or  it « 

1  _ \A/ith 


never  compete  with 
short-term  customer 
demands 

□  Consider  every 
practice  from  PP  and 
PMC  for  your  PI  initiative 

□  Estimate  like  a  pro, 
instead  of  pulling 
numbers  out  of... 


See  also:  “Unplanned  process 
improvement  is  wishful  thinking,’’ 

Watts  Humphrey 

1 
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Be  Right 


Leading 

-=Edger 

PROCESS 
CONSULTANTS  z_z_c 


9.  You  May  Lose  People...  and  That’s 
Okay 

8.  “And  Then  a  Miracle  Happens”  Is  Not 
a  Deployment  Plan 
7.  Process  Improvement  and 

Compliance  Are  Not  Synonymous 
6.  Keep  an  Open  Mind 
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Leading 


Review 


Copyright  2010  Leading  Edge  Process  Consultants  LLC  WWW.CmmiTrainmg.com  2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You 

*  * 

References 


PROCESS 
CO NSU  LTANTS  LL.C 


•  Smith,  Bill.  My  life  (Pittsburgh  PA  metro  area,  Washington  DC  metro  area,  et  al: 


1961  to  date). 

•  Chrissis,  Mary  Beth,  et  al. 

(Boston:  Addison-Wesley,  2003). 

•  Garcia,  Suzanne,  and  Turner,  Richard. 

(Boston:  Addison-Wesley,  2007). 

•  Gerrard,  Matthew,  and  Nevil,  Robbie. 

Miley  Cyrus  as  “Hannah  Montana.”) 

•  Humphrey,  Watts  S. 

•  Martin,  Roger. 

December  1993. 

•  Nietzsche,  Friedrich. 

•  Nonaka,  Ikujiro. 

December  1991. 

•  Senge,  Peter,  et  al. 

(New  York:  Doubleday  1999). 


(song).  (Walt  Disney:  2007).  (Popularized  by 

(New  York:  Addison-Wesley,  1990). 

Harvard  Business  Review,  November- 

1889. 

Harvard  Business  Review,  November- 


Copyright  2010  Leading  Edge  Process  Consultants  LLC 


www.CmmiTraining.com 


2010  NDIA  CMMI  Technology  Conference:  What  Doesn’t  Kill  You...  56 


y 

Week 


er  Presentations 


— 


-  M 


> 


PROCESS 
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...I  hope 
you  were 
able  to 
make  it! 
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Questions 


Intro  to  CMMI  vl  .3 
Hard  Rock  Hotel 
Las  Vegas! 

Feb  8-10,  2011 


Leading 


Tntro  to  CMMI-DEV  vl.3  ^ 

-  Jan  19-21, 2011,  Reston,  VA 
^Feb  8-10,  ,NV> 


Website: 

Blog: 

Alumni  Club 

Twitter: 

YouTube: 

Linkedln: 

Facebook: 


www.  .com 

www.  .com  Bill  Smith 

.ning.com  bill@cmmitraining.com 

twitter.com/ 

www.youtube.com/ 

www.linkedin.com/in/ 

www.facebook.com/pages/ 


120351941356797 


ig.com  2010  NDIA  CMMI  Technology  Conf 
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CMMI  Technology  Conference  and  User 

Group 


NASA  Experience  with 
CMM  and  CMMI 


November,  2010 
John  C.  Kelly  &  Tim  Crumbley 
Office  of  Chief  Engineer 


•  Outline 


NASA  Experience 
with  CMM  and  CMMI 


-  NASA’s  experience 
with  CMMI  model 

-NASA’s  CMMI 
requirement 

-  NASA’s  lessons 
learned  and  key 
impacts  from  using 
CMMI 

Atop  twin  columns  of  fire,  space  shuttle  Atlantis 
roars  into  the  cloudy  sky  above  Launch  Pad  39A  at  NASA's 
Kennedy  Space  Center  in  Florida  on  the  STS-125  mission. 


The  Three  Elements  of  Project 

Success 


Process: 

a  defined  method  involving  steps  or  operations 


People: 

Skills,  Training, 
Management 


E3 

Technology: 
Application  domains, 
tools,  languages, 
information,  environments 


Improved  Process  +  Competent  Workforce  +  Appropriate  Technology 
Reduced  Risk,  Higher  Productivity,  and  Better  Quality 


Timeline  2000-2011 
NASA  Software  Engineering  Initiative  -  CMM  and  CMMI  Activities 


Planned  for 
FY11 


CMMI  Level  3 


MSFC 

JPL 


LaRC* 

(FSSB) 


LaRC* 

(SDAB) 


CMMI  Level  2 


GSFC 

MSFC 

LaRC*  SDAB 
LaRC*  ASDC 

GSFC 

LaRC 

(FSSB) 


JSC 

KSC 

LaRC 


CMM  Level  3 


MSFC 

ARC* 


JSC 


CMM  Level  2 

LaRC 

JPL 

ARC 

GRC 

JPL 

JPL 

JSC 

*-  Implemented 
part  of  the  model 

t 

MSFC 

JSC 

2000 

2001 

2002 

2003 

2004 

2005 

2006  2007  2008  2009  2010 

y 


Software  Working 
Group  Charter 


Started:  NASA  SW 
Eng  Improvement 
Initiative 


Signed:  Software 
Procedural 
Requirement 
NPR  7150.2 

NPR  7150.2  (2004) 


5  Centers  had 
experience 
using  the  models 


Signed:  Software 
Procedural 
Requirement 
NPR  7150.2A 


8  Centers  have 
experience 

using  the  CMMI  model 


Class  A  and  B  -  CMMI  Level  2  or 
CMM  Level  3 

Class  C  -  Center  Decision 


NPR  7150.2  A  (2009) 

Class  A  -  CMMI  Level  3 
Class  B  -  CMMI  Level  2 
Class  C  -  Center  Decision 


CMM/CMMI  Appraisals  &  Consultations  at  NASA  Centers 
Software  Engineering  Training  and  SEI  Training  at  NASA  Centers 


NASA  CMMI  Summary 

Completed  SW  Engineering  Appraisals  from  FY07-FY10 


CMMI  =  Capability  Maturity  Model  Integrated  (Carnegie  Mellon  University  -  Sl/l/  Engineering  Institute) 


Center/Organization 

Rating  (SCAMPI  A  by 
Certified  Appraiser) 

Date 

# 

Projects 

Type 

Organizational 

size 

Software  Classes 
Assessed 

LaRC-  ASDC 

PP(CL3),  CM(CLl) 

Nov-06 

1 

Data  Center  Support 

85 

Class  C 

MSFC 

ML3 

Apr-07 

3 

Development 

63 

Class  A,  B  and  C 

JPL 

ML3 

Sep-07 

7 

Dev  &  Maintenance 

1000 

Class  A,  B  and  C 

GSFC 

ML2  +  RSKM(2) 

May-08 

4 

Dev  &  Maintenance 

600 

Class  A,  B  and  C 

LaRC-  FSSB 

ML2  +  CL3 

Oct-08 

3 

Services 

5 

Class  B  &  C 

LaRC-  SDAB 

PP(CL3),  REQM(CL3), 
CM(CL3),  MA(CL3) 

Mar-09 

4 

Development 

21 

Class  B  &  C 

JSC 

ML2 

Apr-09 

4 

Development 

90 

Class  A,  B,  C  and  D 

KSC 

ML2 

Sep-09 

1 

Development 

225 

Class  A,  B  and  C 

MSFC  -  SIL 

ML2  +  CL3 

May-2010 

1 

Development 

50 

Class  C 

ARC  -ISD 
(Code  Tl) 

ML2 

May-2010 

6 

Development 

63 

Class  B  &  C 

GRC-FIt  SW 

ML2 

Aug  2010 

2 

Development 

22 

Class  C  &  D 

MSFC-FIt  SW 

ML3 

Aug  2010 

1 

Development 

75 

Class  A 

JPL  -  Mission  SW 

ML3 

Sept  2010 

9 

Development  & 
Maintenance 

950 

Class  B  and  C 

& 

Sample  of  NASA  Industry  Partner  Ratings 


NASA  Projects 

Industry  Partners 

CMMI 

Level 

Shuttle  Primary  Avionics  Software  System  (PASS),  Shuttle  SAIL  test 
facility,  Orion  Crew  Exploration  Vehicle  (Orion) 

United  Space  Alliance  Flight  Software  Element  (FSWE) 

Level  5 

International  Space  Station  (C&DH),  Ares 

Boeing 

Level  3 

Orion  Crew  Exploration  Vehicle  (Orion) 

Lockheed  Martin  Corporation 

Level  3 

Orion  Crew  Exploration  Vehicle  (Orion),  Ares 

Honeywell 

Level  3 

Orion  Crew  Exploration  Vehicle  (Orion) 

ATK 

Level  3 

Ares 

Draper 

Level  3 

Ares  J-2X,  Orion  Crew  Exploration  Vehicle  (Orion),  Space  Shuttle 

Main  Engine 

Hamilton  Sundstrand  Rocketdyne  /  Pratt  &  Whitney 
Rocketdyne 

Level  3 

Ares,  Deep  Impact  1 

Ball 

Level  3 

James  Webb  Space  Telescope 

Northrop  Grumman 

Level  3 

GRAIL,  Juno 

Lockheed-Martin  Space  and  Exploration  Systems 

Level  3 

Ground  Systems  Engineering  (GSE)  Checkout,  Assembly  and 

Payload  Processing  Services  (CAPPS)  Kennedy  Space  Center  (KSC) 

Boeing 

Level  3 

ISS  Environmental  Control  and  Life  Support  Systems,  Orion  Crew 
Exploration  Vehicle  (Orion) 

Hamilton  Sundstrand 

Level  3 

GOES-R 

Harris  IT  Services  Corporation 

Level  3 

MSFC  Engineering  Support  Contractor 

Jacobs  Engineering 

Level  3 

STEREO 

Johns  Hopkins  University  Applied  Physics  Laboratory 

Level  3 

NASA  Software  IV&V  Services,  JSC  Support  Contractor 

L-3  STRATIS 

Level  3 

KSC  Support\Shuttle  support 

United  Space  Alliance,  LLC 

Level  3 

NASA  Aircraft  Management  Information  System  (NAMIS)  software 

SAIC,  Aircraft  Operations  Support  System  (AOSS) 

Level  3 

JSC  Support  Contractor 

Tietronix 

Level  2 

NASA’s  Software  Engineering  Requirements 

•  Software  engineering  is  a  core  capability  and  a  key  enabling  technology  for 
NASA's  missions  and  supporting  infrastructure. 

•  The  NASA  Software  Engineering  Procedural  Requirements  (NPR  7150.2A) 
supports  the  implementation  of  the  NASA  Policy  Directive  (NPD)  7120.4, 

NASA  Engineering  and  Program/Proiect  Management  Policy. 

•  The  NASA  Software  Engineering  Requirements  provide  a  minimal  set  of 
requirements  established  by  the  Agency  for  software  acquisition, 
development,  maintenance,  retirement,  operations,  and  management. 

•  The  NASA  Software  Engineering  Requirements  are  intended  to  support 
NASA  programs  and  projects  to  accomplish  their  planned  goals  (e.g., 

mission  success,  safety,  schedule,  and  budget)  while  satisfying  their 
specified  requirements. 

•  The  NASA  Software  Engineering  Requirements  provide  a  set  of  software 
engineering  requirements  in  generic  terms  to  be  applied  throughout  NASA 
and  its  contractor  community. 
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Profile  of  NPR  target  audience 


Early 

Adopters 


Progressive 

Users 


Slow 

Adopters 


Entrenched 

Resisters 


Purpose  of  NPRs 


Shift  target  audience  to 


the  left  10  -  25% 


This  is  our  target  after  putting  the  NPR  in  place  and  after  each  NPR  update  cycle 
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<s 

NASA-\ 

/vide  Software  Classification* 

Class  A 

Space  Flight  Human  Rated  Software  Systems 

Class  B 

Non-Human  Space  Rated  Software  Systems 

LU 

O 

O 

Class  C 

Mission  Support  Software  &  Facilities 

Class  D 

Analysis  and  Distribution  Software 

Class  E 

Development  Support  Software 

(e.g.,  Class  A  -  C  is  mostly  software  developed  or  acquired  for  Highly  Specialized  IT  systems) 

Class  F 

General  Purpose  Computing  Software 

n 

(Multi-Center  or  Multi-Program/Project) 

u 

O 

Class  G 

General  Purpose  Computing  Software 

(Single  Center  or  Project) 

^Class  H 

General  Purpose  Desktop  Software 

*  Established  by  NPR  7150.2A 

10 

NASA  Software  Engineering  Requirements,  NPR  7150.2 

CMMI  Requirement 

•  [SWE-032]  The  project  shall  ensure  that  software  is  acquired,  developed  and 

maintained  by  an  organization  with  a  non-expired  Capability  Maturity  Model 
Integration®  for  Development  (CMMI-DEV)  rating  as  measured  by  a  Software 
Engineering  Institute  (SEI)  authorized  or  certified  lead  appraiser  as  follows: 

-  For  Class  A  software:  CMMI-DEV  Maturity  Level  3  Rating  or  higher  for  software, 
or  CMMI-DEV  Capability  Level  3  Rating  or  higher  in  all  CMMI-DEV  Maturity  Level 
2  and  Maturity  Level  3  process  areas  for  software. 

-  For  Class  B  software:  CMMI-DEV  Maturity  Level  2  Rating  or  higher  for  software, 
or  CMMI-DEV  Capability  Level  2  Rating  or  higher  for  software  in  the  following 
process  areas: 

•  a.  Requirements  Management. 

•  b.  Configuration  Management. 

•  c.  Process  and  Product  Quality  Assurance. 

•  d.  Measurement  and  Analysis. 

•  e.  Project  Planning. 

•  f.  Project  Monitoring  and  Control. 

•  g.  Supplier  Agreement  Management  (if  applicable). 


-  For  Class  C  software:  The  required  CMMI-DEV  Maturity  Level  for  Class  C 
software  will  be  defined  per  Center  or  project  requirements. 
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CMMI  Requirement  Notes 

•  Note:  Organizations  who  have  completed  Standard  CMMI®  Appraisal  Method  for 
Process  Improvement  (SCAMPISM)  Class  A  appraisals  against  the  CMMI-DEV  Model 
are  to  maintain  their  rating  and  have  their  results  posted  on  the  SEI  web  site  so  that 
NASA  can  assess  the  current  maturity/capability  rating.  Software  development 
organizations  need  to  be  reappraised  and  keep  an  active  appraisal  rating  posted  on  the 
SEI  web  site  during  the  time  that  they  are  responsible  for  the  development  and 
maintenance  of  the  software. 

•  Note:  For  Class  A  software  development  only,  a  transition  period  to  obtain  a  CMMI- 
DEV  Maturity/Capability  Level  3  Rating  will  be  allowed  for  organizations  developing 
Class  A  software  per  the  NASA  Headquarters  Office  of  the  Chief  Engineer  approved 
Center  Software  Engineering  Improvement  Plan  as  described  in  SWE-003,  SWE-004, 
and  SWE-108. 

•  Note:  For  Class  B  software,  in  lieu  of  a  CMMI  rating  by  a  development  organization, 
the  project  will  conduct  an  evaluation,  performed  by  a  qualified  evaluator  selected 
by  the  Center  Engineering  Technical  Authority,  of  the  seven  process  areas  listed  in 
SWE-032  and  mitigate  any  risk,  if  deficient.  This  exception  is  intended  to  be  used  in  those 
cases  in  which  NASA  wishes  to  purchase  a  product  from  the  "best  of  class  provider",  but 
the  best  of  class  provider  does  not  have  the  required  CMMI  rating.  When  this  exception 
is  exercised,  the  Center  Engineering  Technical  Authority  should  be  notified. 


CMM/CMMI  Lessons  Learned  by  NASA 


Preparing  for  an  appraisal  is  where  you  get  the 
measurable  process  improvement 

CMMI  process  helped  Centers  establish  a  baseline 
of  where  they  are 

Develop  an  extensive  set  of  “tools”  (i.e.,  templates, 
spreadsheets)  to  help  projects  with  CMMI  practices 
and  artifacts 

-  Use  of  toolset  helped  projects  reach 
compliance  much  faster 

Mentors  can  help  get  Project  tool  use  started  and 
help  Projects  tailor  the  artifacts 

Established  sponsorship  across  departments 

-  Management  Steering  Group 

-  Was  difficult  to  get  mid-level  managers  to 
“own”  improvement  program 

Established  early  on  a  relationship  with  the  Lead 
Appraiser 

PHD  development  and  artifact  collection 

-  PIIDs  and  artifacts  were  maintained  on  a 
server  for  ease  of  access  and  review 


The  perfectly  picturesque  spiral  galaxy 
known  as  Messier  81,  or  M81 


Importance  of  interview  preparation  and  training 
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CMM/CMMI  Lessons  Learned  by  NASA 


Use  the  workshops  to  review  the  processes  in 
depth  and  reinforced  the  tool  sets 

Tracking  Progress,  determine  a  method  for 
projects  to  report  progress 

Many  of  our  projects  need  basic  project 
management  and  configuration  management 
training 

CMMI  assessments  helped  identify  areas  for 
process  and  project  improvement 

Despite  initial  reluctance,  pre-appraisal  was  a 
positive  experience  for  our  projects  -  laid  a  good 
foundation  for  future  involvement 

Projects  appreciated  systematic  and  analytical 
feedback  on  what  they  are  doing 

Measurement  and  analysis  is  a  big  challenge 

Improved  quality  and  review  of  management  plan 
early  in  the  life  cycle  and  reuse  of  the  plans  for 
new  projects 

Resource  planning  and  tracking  at  the  individual 
process  level  provided  little  additional  benefit  to 
the  projects 


Mars 

Image  Credit:  NASA/JPL-Caltech 


Smaller  projects  need  to  have  light-weight 
processes  to  avoid  being  smothered  (especially 
for  a  one  person  task) 
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CMMI  Key  Impacts  at  NASA 


Space  Shuttle  Launch  picture 
from  Disney’s  Castle 


Reduces  risk  of  software  failure  -  Increases  mission  safety 

-  Improvement  processes  based  on  best  practices  in  Industry  and 
Government 

-  Risk  management  much  improved  on  software  subsystems-Previously 
there  was  little  monitoring  of  risks 

More  predictable  software  cost  estimates  and  delivery  schedules 

-  Data  showed  projects  working  within  CMMI  software  framework  &  best 
practices  had  increased  accuracy  in  cost  estimates  and  smaller  growth  in 
resources  over  the  lifecycle 

Smarter  buyer  of  contracted  out  software 

-  Educating  the  NASA  workforce  on  best  practices  in  Software  Engineering 

More  defects  found  and  removed  earlier 
Reduces  duplication  of  efforts  between  projects 

Increases  ability  to  meet  the  challenges  of  evolving  software 
technology 

Software  development  planning  has  been  improved  across  the 
Agency 

-  There  is  a  growing  consensus  among  the  practitioners  and  software 
managers  that  working  to  a  defined  process  has  substantial  benefits. 

-  Vast  improvement  in  planning  of  software  projects  and  in  monitoring 
progress 

NASA’s  contractor  community  has  heard  the  word  that  the  bar 
has  been  raised  with  respect  to  software  engineering  and  is 
responding  appropriately 

-  NPR  7150.2A,  Software  Engineering  Requirements  (update  Nov  2009) 
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CMMI  Key  Impacts  at  NASA 


A  solid  foundation  and  structure  is  now  in  place 
for  developing  software  in  a  disciplined  manner 

More  uniformity  in  management  plans,  reviews,  test 
plans,  status  reporting  Risk  management  much 
improved  on  software  subsystems-Previously  there 
was  little  monitoring  of  risks 

Data  management  and  configuration  management  has 
improved 

Improve  the  working  relationships  between  Engineering 
and  Safety  and  Mission  Assurance  with  respect  to 
software  engineering 

The  Agency  is  better  prepared  for  major 
programs  and  projects  than  it  was  8  years  ago 

Software  teams  and  software  quality  engineers  are 
working  together  to  assure  compliance  to  standards,  to 
improve  quality 

The  knowledge  and  skills  of  the  NASA  software 
engineering  community  has  significantly 
improved 

We  have  seen  significant  cultural  changes. 

Extensive  mentoring  program  established  to 
improve  software  practices 

Our  projects  are  now  better  managed  - 
particularly  in  the  area  of  progress  tracking 

Now  we  know  exactly  where  we  are  in  the  project  and 
how  long  it’s  likely  to  take  to  finish 
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Space  Shuttle  Atlantis  Rollout  at 
Kennedy  Space  Center 
April  2010 


Summary  of  NASA’s  Experience 
with  CMM  and  CMMI 

Why  improve  processes?  Because  process  is  the  foundation  for  all  other  improvements,  and  lasting 
improvements  are  not  possible  without  it. 

If  a  performance  management  system  is  not  in  use,  leadership  is  unaware  of  what  is  and  is  not  working. 

CMMI  is  a  proven  approach  to  performance  management  -  with  more  than  a  decade  of  results  showing 
it  does  work. 

Simply  deciding  to  “do  CMMI”  is  not  enough 
to  achieve  benefits. 

Defining  good  processes,  using  them,  measuring 
the  results,  and  making  improvements  based  on 
what  you  learn  are  all  key  to  reaping  the  benefits 
of  process  improvement. 

The  CMMI  models  are  one  part  of  a  comprehensive 
approach  to  process  improvement  that  helps 
organizations  understand 

•  why  they  should  improve 

•  what  frameworks  and  tools  would  best  fit  their 
needs 

•  how  to  implement  them  Apollo  8,  the  first  manned  mission  to  the  moon 
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Why  CM? 


■  CM  ensures  that  the  current  configuration  of  items  are 
known  throughout  their  lifecycle 

■  CM  ensures  that  changes  to  the  configuration  of  evolving 
items  are  correct,  controlled,  managed,  and  documented 

■  CM  helps  manage  complexity,  interface  dependencies, 
increases  security,  and  recovery  from  errors 
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What  is  CM? 


■  CM  is  a  discipline  applying  technical  and  administrative 

direction  and  surveillance  to: 

Identifying  and  documenting  the  physical,  functional,  and 
performance  characteristics  of  items 

-  Baselining  those  characteristics 

-  Controlling  changes  to  those  characteristic 

-  Providing  status  on  those  characteristics 

-  Conducting  audits  on  those  characteristics 

■  The  CM  tasks  that  produce  these  results  are: 

Configuration  Planning 

-  Configuration  Identification 

-  Configuration  Control 

-  Configuration  Status  Accounting 

-  Configuration  Management  Audits 
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Application  of  CM 


■The  CM  concepts  presented  in  this  course  can  be 
applied  to: 

-  Hardware  (H/W) 

-  Software  (S/W) 

-  Facilities 

And  their  appropriate  documentation 


During  Development  and  Operation  by  the 

Acquirer  and  Supplier 
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Some  Levels  of  CM 
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Configuration  Management  Overview 


Configuration  Management  Audits  -  Configuration  Status 
Accounting _ 


8 


MITRE 


©2009  The  MITRE  Corporation.  All  rights  reserved. 


Configuration  Identification  continued 


■  Three  level  of  Configuration  Identification  are 
established 


Functional  Configuration  Identification  (FCI) 


Allocated  Configuration  Identification  (ACI) 


Physical  Configuration  Identification  (PCI) 


Hardware 
Conceptual  Software 

Systems  Facilities  Design  Implementation  Test  Operation 


Requirements 

Requirements 

FCI  ACI 


Lifecycle  Phases 


PCI 
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Functional  Configuration  Identification 

Functional  Configuration  Identification  (FCI) 

The  identified  system  and  system  items  and  their 
physical,  functional,  and  performance  characteristics 
which  are  documented  in  a  System  Specification 


System  Cl 
Specification 
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Allocated  Configuration  Identification 

Allocated  Configuration  Identification  (ACI) 

Later  in  development  the  physical,  functional,  and 
performance  characteristics  of  the  system  are 
allocated  to  lower  level  entities:  software,  hardware, 
facilities,  and  are  documented  as  Allocated 
Specifications  for  requirements 
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Physical  Configuration  Identification 


Physical  Configuration  Identification  (PCI) 

Finally,  the  products  of  the  developed  system: 
software,  hardware,  facilities  are  defined  in  a  series  of 
Product  Specifications  that  describe  the  as-built 
system 
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Formal  Baselines 


Baselines  are  established  at  strategic  points  in  a 
system  lifecycle.  Three  baselines  may  be  defined 

■  Functional  Baseline  (FBL) 

■  Allocated  Baseline  (ABL) 

■  Product  Baseline  (PBL) 


Systems 

Requirements 

FCI  -  Cl 

\ 

Hardware 
Software 
Facilities 
Requirements 
ACI  -^CIs 

Design 

Implementatio 

n  Test 

PCI  - 

Operation 

CIs 

/ 

X 

FE 

3L  AE 

3Ls 

PB 

1 

Ls 

Lifecycle  Phases 
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Configuration  Identification  and 
Configuration  Items 


■  Configuration  Identification  is  an  activity  that  identifies 
items  and  their  characteristics:  physical,  functional,  and 
performance 

■  Not  all  items  that  are  identified  need  be  controlled  at  the 
same  level  of  rigor 

■  Configuration  Items  are  selected  for  formal  change  control 
from  items  identified 


Configuration  Identification  -  Software 


*Operating 

*Network 

**Navigation 

^Communication 

**Test 

System 

Software 

Software  Cl 

; 

Software  Cl 

Software  Cl 

Commercial  products  MAY  not  be  subject  to  change  -  In  operation  everything  is  under  CM 
control 

^Applications  software  in  development  that  is  subject  to  change 
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Configuration  Item 


Represents  the  characteristics  of  a  Configuration  Item 
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Baseline  vs.  Configuration  Items 


■  The  approved  and  fixed  (baselined)  configuration 
of  a  Cl  at  a  specific  time  in  its  lifecycle  that  serves 
as  a  reference  point  for  change  control 


CIs  are  used  for  visibility 

i — t — i 

Cl 


Visibility 


Baselines  are  used  for  control 


Control 
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Configuration  Control 

■  The  systematic 

-  evaluation 

-  coordination 

-  approval  or  disapproval,  and 

-  implementation 

of  changes  to  the  physical,  functional,  and 
performance  characteristics  of  a  baselined  Cl 

■  Changes  are  requested  with  a  Change  Request 
(CR)  form 
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Configuration  Control  Board  (CCB) 


■  Establishes  baselines  for  CIs 

■  Reviews  and  approves  /  disapproves  /  defers  Change 
Requests  to  CIs 

■  Membership  comprised  of  management,  and  other 
stakeholders  and  supported  by  the  subject  matter 
experts 

-  Project  Management 

-  Systems  Engineering 

-  Software/Hardware  Engineering 

-  Test  Engineering 

-  Quality  Assurance 
Configuration  Management 

■  Chaired  by  the  program  /  project  manager  or 
designee 
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Technical  Review  Board  (TRB) 


■  Provides  technical  and  programmatic  support 
to  the  CCB 

-  Conducts  impact  assessment  on  CRs  to 
baselined  CIs 

-  Makes  approval  /  disapproval  recommendations 
to  the  CCB 

■  Membership  comprised  of  program  /  project 
personnel  and  subject  matter  experts 

■  Chaired  by  a  technical  manager 
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CCB  and  TRB  Hierarchy 
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Configuration  Control 


Configuration  Item 


Less  than 
3  mph  wind 


Physical  characteristic 


Functional  and  performance  characteristic 


Constraint 

\ 

3%  Grade 


Gravel 


Change  Request 


Need  to  control  the  configuration  of  physical, 
functional,  and  performance  characteristic 


If  not  we  might  get  something  really  dumb 
or  suffer  a  catastrophic  failure 
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CR  Example 


Change  Request 


CR# 

Date:  12/4/2003 

Requestor:  ET 

Class:  |  □  ||  □ 

Problem:  A  requirement  to  deploy  the  probe’s  parachute  does  not  exist 

Change:  Add  the  following  requirement:  The  probe’s  parachute  shall  be  deployed  .01  second: 
after  the  heat  shield  has  been  jettisoned 

Impacts:  Enter  figures  for  cost  and  schedule  and  list  affected  interfaces  or  “None”  and  attach 
impact  assessments 

Systems: 

Hardware: 

Software: 

Test: 

Configuration  Management: 

Quality  Assurance: 

Contracts: 

Other  [Specify]: 

Approve: 

TRB  Date: 
CCB  Date: 

Chair: 

Chair: 

Disapprove: 

TRB  Date: 
CCB  Date: 

Chair: 

Chair: 

Assignee: 

Due  Date: 
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Change  Flow 


Request  Change  (CR) 


Evaluate  Change 


Approve  Change 


Implement  Change 


Track  Change 


MITRE 


Supplier  or  Acquirer 


TRB 


CCB 


Owner  of  item 


CM  staff  and  owner  of  item 
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Impact  Assessments 


■  Impact  assessments  need  to  be  conducted  by  all  stakeholders: 

-  Systems 

-  Hardware 

-  Software 

-  Test 

-  Configuration  Management 

-  Quality  Assurance 

-  Contracts 

-  Others 

■  On  Cl  characteristics: 

Physical 
Functional 

-  Performance 


■  Against  their  interests: 

-  Cost 

-  Schedule 

-  Interface 
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Classification  of  Changes 


At  least  two  types  of  changes  can  be  defined: 

■  Class  I — affects  the  Acquirer’s  interest  in  one  or  more  of 
these  factors: 

-  Physical  characteristics 

-  Functional  capability 
Performance 

-  External  interfaces 

-  Cost 

-  Schedule 


Supplier  must  submit  change  to  the  Acquirer 
for  approval  before  implementation 
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Classification  of  Changes  concluded 


■  Class  II  -  Does  not  affect  any  of  the  Class  I  factors 

affects  changes  such  as: 

-  Spelling  or  typographical  errors 

-  Addition  of  clarifying  comments 

-  Changes  that  do  not  affect  external  interfaces,  change 
functionality  or  degrade  performance 


Supplier  may  implement  it  without  Acquirer’s  approval 
but  must  inform  Acquirer  of  change 
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CM  Audits 


■  Functional  Configuration  Audits  (FCA)  and  Physical 
Configuration  Audits  (PCA)  are  conducted  by  Engineering 
and  facilitated  by  CM  and/or  Quality  Assurance  (QA) 

■  Other  audits  conducted  by  QA  and  CM  may  include: 


Audits  of  CM  Repository  that  contains  CM  records,  documentation, 
processes,  procedures,  artifacts,  etc. 


Audits  of  Program/Project  organizations  to  ensure  CM  process  is  being 
followed 


Audits  of  status  of  approved  CRs 
Audits  to  ensure  that  CIs  are  consistent 
with  CM  records 


PCA  can  be  prolonged 
until  after  Operational  Tests 
If  necessary 


Conceptual 

Systems 

Requirements 


Hardware 

Software  Design  Implementation  Test  Operation 

Requirements 


FCA/PCA 
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Functional  Configuration  Audit  (FCA) 

■A  formal  examination  of  test  results  of  the  as-built 
functional  configuration  of  CIs,  prior  to 
acceptance,  to  verify  that  the  CIs  have  satisfied 
their  specified  requirements 

■This  audit  is  conducted  by  the  Supplier  for  the 
Acquirer  and  attended  by 

-  Management 

-  System  Engineering 

-  Hardware  /  Software  Engineering 

-  Test  Engineering 

-  QA  and  CM 

-  Contracts 

of  both  the  Acquirer  and  Supplier 
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Functional  Configuration  Audit 
continued 


Functional 


Testing 


Requirements 
Specifications 
Requirements 
Traceability 
Test  Plans 
Test  Scenarios 


>- 


Test  Results 


Physical  Configuration  Audit 
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Physical  Configuration  Audit  (PCA) 


■A  formal  examination  of  the  as-built  physical 
configuration  of  Cl  products  against  their  design 
documentation 

■This  establishes  the  Product  Baseline 

■This  audit  is  conducted  by  the  Supplier  for  the 
Acquirer  and  attended  by 

-  Management 
-System  Engineering 

-  Hardware  /  Software  Engineering 
-Test  Engineering 

-  QA  and  CM 

-  Contracts 

of  both  the  Acquirer  and  Supplier 
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Physical  Configuration  Audit 
continued 


Design 

Documentation 


Implementation 


J 


Physical 


Supplier  As-Built  Products: 

•  Design  Documentation 

•  Code 

•  Hardware 

•  Etc. 


Inputs 


> 


Physical 

Configuration  Audit 

Examination  of  the  “as-built” 
configuration  of  CIs  against  their 
documentation 

Supplier 

Acquirer 


Outputs 


Product  Baselines 
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Configuration  Status  Accounting 
(CSA) 


■CSA  is  performed  to  gather,  correlate,  maintain 
and  provide  status  on  controlled  products  (CIs), 
and  on  CM  tasks 


CM  Planning 


f 

Configuration 

Identification 

Configuration 

Control 

< 

Configuration 

Audits 

1 

Configuration 

V  : 

Status  Accounting 

Products  (CIs) 


CM  Tasks 
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Configuration  Status  Accounting 
continued 

■The  Configuration  Status  Accounting  (CSA) 
task  gathers,  correlates,  maintains,  and 
provides  status  on  CM  controlled  products 
and  CM  tasks 

■  Provides  the  means  for  reporting  status  on: 
-  Configurations  -  Baselines  -  Other 


■  FCI 

■  ACI 

■  PCI 


FBL 

ABL 

PBL 


CM  metrics 
CM  activities 
CM  Audits 
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Configuration  Status  Accounting 
concluded 


Supplier 


Configuration 

Status  Accounting 
Reports 
produced  by 
the  CM  organization 

r 

Management  and  Staff 

Acquirer 


Reviews 
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Presentation  Contents 

■  Introduction 

-  Reasons  for  Configuration  Management  (CM) 

-  CM  Concepts 

■  Formal  CM 

-  Formal  Baselines  and  Configuration  Items  (CIs) 

-  Configuration  Control  Boards  (CCBs) 

■  Supported  with  Technical  Review  Boards  (TRBs) 

-  Change  Control 

-  CM  Audits  and  Status  Accounting 
Internal  CM 

-  Internal  Baselines 

CM  of  Design,  Code,  Hardware  Items,  Test  Articles 

■  Operation  CM 

During  Operation  /  Maintenance 

■  References 
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Internal  CM  versus  Formal  CM 


■  Formal  CM  is  concerned  with 

-  High  Level  baselines 

.  FBL 
.  ABL 
.  PBL 

-  Master  Schedules 

-  Contractual  Items 

■  Internal  CM  is  concerned  with 

-  Design  BL 

-  Code  BL 

Hardware  component  BL 

-  Test  BL 

-  COTS  BL 

-  Etc. 
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Internal  CM  Concerns 


■  Documents 

-  Database 

-  Test  procedures 

Analysis  that  drive  requirements  and  design 
—  Etc. 

■  Plans 

-  Project  plans 

-  CM  plans 

-  QA  plans 

-  Risk  Management  plans 

-  Test  plans 
—  Etc. 
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Formal  CM  Under  Configuration  Control 
Board  (CCB) 

■  Configuration  Control  Board  is  Chaired  by  PM 

■  Membership  composed  of  management 

-  Systems 

-  Software 

-  Hardware 
-Test 
-CM 
-QA 

-  Etc. 
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Internal  CM  Under  Technical  Review 
Board  (TRB) 

■  Chaired  by  Deputy  PM  or  Lead  Systems  Engineer 

-  Systems 

-  Software 

-  Hardware 
-Test 
-CM 
-QA 

-  Etc. 


39 


MITRE 


©2009  The  MITRE  Corporation.  All  rights  reserved. 


Internal  CM  Concerns  continued 


■  Internal  CM  is  concerned  with 

-  Version  Control 

■  Documents 

■  Code 

■  Hardware  items 
.  COTS 

-  Data  Management 

■  Documents 

■  Plans 

■  Process  Documentation 

■  Procedures 

■  Metrics 

■  Action  Items 

■  Etc. 
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Internal  CM  &  Testing 


■  Internal  CM  during  testing  is  concerned  with 

-  Code  changes  (TRB) 

-  Design  changes  (TRB) 

-Test  case  changes  (TRB) 

-  Requirements  changes  (Require  escalation  to  CCB) 
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Internal  Baselines 


Internal  baselines  are  established  at  strategic  points 
in  a  system  lifecycle.  Three  internal  baselines  may 
be  defined 

■  Design  Baseline  (DBLs) 

■  Code/Hardware  Components  Baseline  (C/HCBLs) 

■  Test  Baseline  (TBLs) 


Hardware 

Software 


Systems 
Requirements 
FCI  -  Cl  AC 
\  Cl 

Facilities 

Requirements 

;i- 

s 

Design 

Implementatio 

n  Test 

PCI  - 

Operation 

CIs 

\ 

i 

FBL 


ABLs 


DBLs  C/HCBLs  PBLs 

TBLs 


Lifecycle  Phases 
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Internal  CM  During  Design 

Design  not  yet  Baselined 
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Internal  CM  During  Coding 

Design  Baselined,  Code  not  Baselined 
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Internal  CM  During  Testing 

Design,  Code  &  Test  Cases  Baselined 
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CM  During  Operation 


■  Operation  CM  does  not  differ  from  CM  conducted  during 
development 

-  Formal  CM 

-  Internal  CM 

■  The  players  may  change 

-  A  different  Operation  contractor 

-  A  different  Operation  agency 

■  Acquisition  Agency  vs.  Operation  Agency 

■  The  Operation  Baseline  has  been  established 
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Presentation  Contents 

■  Introduction 

Reasons  for  Configuration  Management  (CM) 

-  CM  Concepts 

■  Formal  CM 

Formal  Baselines  and  Configuration  Items  (CIs) 

-  Configuration  Control  Boards  (CCBs) 

■  Supported  with  Technical  Review  Boards  (TRBs) 

-  Change  Control 

-  CM  Audits  and  Status  Accounting 

■  Internal  CM 

-  Internal  Baselines 

CM  of  Design,  Code,  Hardware  Items,  Test  Articles 

■  Operation  CM 

During  Operation  /  Maintenance 

■  References 
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CM  During  Operation  continued 


Defects  and  changes  during  Operation  may  require 
repeat  of  activities  that  were  conducted  during 
development  and  reestablishment  of  baselines  as 
appropriate. 
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References/Suggested  Reading 


IEEE  Std.  828-1998  IEEE  Standard  for  Software  Configuration 
Management  Plans 

m  IEEE  1042,  Guide  to  Software  Configuration  Management 

m  ANSI/EIA-649- 1 998  National  Consensus  Standard  for 
Configuration  Management 

m  IEEE  828-2005  -  Standard  for  Software  CM  plans 

m  MIL-STD-973  Military  Standard  for  Configuration  Management 
(cancelled,  but  still  good  reference) 

m  CM  Today  Yellow  Pages,  Your  Source  for  Daily  CM  News, 
www.cmtoday.com/yp/configuration_management.html 

■  CM  BoK  -  Configuration  Management  Body  of  Knowledge. 
www.cmcrossroads.com/cgi-bin/cmwiki/bin/view.cgi/CM/ 
CMBoK,  CM  Crossroads,  CM  Community  Forums 

■  Capability  Maturity  Mode  Integration  (CM Ml®),  Version  1.3 
Software  Engineering  Institute 
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Contact  Information 


Al  Florence 
florence@mitre.org 
703  983  7476 
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STRENGTH  T1IK01G1I INDUSTRY  &  TECHNOLOGY 


Using  CMMI®  Effectively 
for  Small  Business 

—  Panel  — 

(With  interactive  discussion  from  panel  and  audience  recorded  in  slides) 


NDIA  CMMI®  Working  Group 
NDIA  Systems  Engineering  Division 

2010  CMMI  Technology  Conference 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
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Is  CMMI  for  Small  Business? 


STRENGTH  TIIKOIGII 1NDLSTRV  &  TECHNOLOGY 


The  NDIA  CMMI  Working  Group  was 

tasked  to  investigate  common  questions 

on  the  applicability  of  CMMI  to  small 

business 

•  Is  CMMI  appropriate  for  small  businesses? 

•  Is  CMMI  a  barrier  to  competition  in  the 
defense  market? 

•  What  data  substantiates  the  adoption  of 
CMMI  by  small  business? 

•  What  challenges  do  small  businesses  face 
for  adoption? 

•  How  can  acquirers  use  CMMI  to  manage 
their  acquisition  risks  with  small  suppliers? 

Reference:  ‘CMMI  for  Small  Organizations’,  NDIA  CMMI  Working  Group,  CMMI  Conference,  Nov  2010. 

Panel  discussions  will  be  recorded  in  this  slide  set,  as  applicable. 
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Panelists  - 

Using  CMMI  Effectively  for  Small  Business 

Lynn  Penn 
Lockheed  Martin 

Girish  Seshagiri 

Advanced  Information  Services  (AIS) 

Rosalind  Singh 
Raytheon  Company 

Dr.  Rick  Welch 
Northrop  Grumman 
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What  advantages  and  obstacles  do  small 
businesses  face  relative  to  CMMI  adoption? 


STRENGTH  TIIKOIGII INDISTRY  &  TECHNOLOGY 


Small  Business  Advantages 

Small  Business  Obstacles 

•Innovation,  speed,  agility 
•Commonality  across  projects,  tailoring? 

•Fewer  communication  paths 

•Simpler  process/tools  (low  tech  solutions)? 

•Easier  institutionalization  across  projects? 

•CMMI:  too  big,  too  complex?  Sometimes  more 
perception  than  reality  -  usually  appears  in  plans 
•Fewer  dedicated  resources  (staff,  $).  Everybody  is 
a  shared  resource,  sharing  time  with  competing 
demands. 

•Roles,  responsibilities  -  multiple  hats  (QA,  Test, 
Process  Group,  training,  ...) 

•Infrastructure/overhead  costs,  amortization? 

•Cultural  resistance  or  bias? 

•Fewer  people  filling  multiple  roles  allows  easier  to 
connect  the  dots 

•Able  to  be  more  nimble,  react  to  changes  more 
quickly 

•Fewer  approvals  and  needed,  greater  coverage. 
•Less  documentation  needed  for  fewer  people  -  1 
page  descriptions  can  suffice.  Set  thresholds, 
entry/exit  criteria,  etc. 

•Alignment  and  communication  of  objectives  is 
straightforward.  Clear  rationale  for  process 
changes. 

•Small  project,  site,  organization 

•Run  out  of  people  to  provide  objective  input.  Must 

procure  external  resources  to  provide  feedback. 

•Can  be  difficult  to  restructure  existing  working 
relationships  -  doing  things  differently.  “Culture  eats 
process  for  lunch” 
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Improving  Processes  in  Small  Settings 
(IPSS) 


Just  enough  process,  just  in  time  to  support  the  needs  in 
small  settings 


Obstacles  were  outweighed  by  the  advantages  -  similar  to 
the  prior  slide 


Integrating  best  practices,  listening,  facilitating 
Defining  the  process  tolerance 
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Supporting  Small  Suppliers 


STRENGTH  THROUGH 1NDLSTRV  &  TK'HNOLOGV 


Acquirers  (such  as  prime  contractors)  often  use  CMMI  practices  to 
select,  manage,  and  support  their  small  suppliers. 


Typical  uses  of  CMMI  within  larger 
companies  when  working  with  smaller 
suppliers: 

•  CMMI  maturity  level  requirements  rarely 
specified  to  subs  unless  levied  by  external 
customer 

•  Use  CMMI  practices  in  key  areas  to  evaluate 
supplier  risks 

•  Checklists  or  tools  used  for  supplier  selection 

•  May  require  suppliers  to  follow  prime 
contractor’s  processes 

•  Provide  direct  support  to  suppliers  -  training, 
mentoring,  tools,  etc. 

•  Customize  areas  of  interest  per  the  situation. 

•  Subcontracting,  partnering 

•  Find  gaps  that  need  to  be  filled  (e.g.,  EVMS, 
federal  contracting) 

•  Smaller  companies  may  not  have  in-house 
expertise  in  some  CMMI  PAs  (e.g.,  HiMat) 


Alignment  of  Processes  with  Subs 


Requirements  Management 

Project  Planning 

Project  Monitoring  &  Control 

Supplier  Agreement  Management 

Measurement  &  Analysis 

Product  &  Process  Quality  Assurance 

Configuration  Management 


CMMI  Process  Areas 
Level  3  (continued) _ 


organizational  rrocess  uetinition 
Organizational  T raining 
Integrated  Project  Management  for  IPPD 
Risk  Management 
Integrated  Teaming 
Integrated  Supplier  Management 
Decision  Analysis  &  Resolution 
Organizational  Environment  for 
Integration _ 


(4) 

(1) 

(1) 

(1) 

(4) 

(1) 

(4) 


Requirements  Development 
Technical  Solution 
Product  Integration 
Verification 
Validation 

Organizational  Process  Focus 

Notes: 


Organizational  Process  Performance 
Quantitative  Project  Management 


Organizational  Innovation  &  Deployment 
Causal  Analysis  &  Resolution  _ _ 


(5) 

(5) 


(1)  Subcontractor  internal  processes  and  IPT  operations  integrate  with  prime’s  processes. 

(2)  System  requirements  are  allocated  by  prime;  subcontractors  develop  requirements  at  the  configuration  item  (Cl)  level. 

(3)  All  subcontractors  integrate,  verify  and  validate  their  products  to  the  Cl  or  subsystem  level;  this  includes  integration  of  software  CIs  into 
hardware  CIs  or  line  replaceable  units  (LRUs).  Prime  integrates,  verifies  and  validates  at  the  system  level 

(4)  Subcontractors  follow  their  own  CMMI-compliant  business  processes. 

(5)  Prime  is  responsible  team’s  process  control  and  optimization. 


jvGfrmnor  gfiummam 


ISER-MLB-PR-07-143 


Reference:  “High  Maturity  System/Software  Cost  Estimation”, 
Richard  L.  W.  Welch,  PhD.,  Northrop  Grumman  Integrated  Systems 
CMMI  Technology  Conference,  November  2007. 
http://www.dtic.mil/ndia/2007cmmi/Wednesdav/4pmWelch.pdf 
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Risk  Based  Subcontractor 

Assessment 

Lockheed  Martin  Corporation 


Developed  Resources 


► 


► 


Created  Risk-Based  Subcontractor  Assessment 
Guidebook: 

°  Includes  initial  and  on-going  evaluations 

°  Defines  assessment  team  roles  and  responsibilities 

°  Explains  how  and  who  to  interview  during  a 
subcontractor  evaluation 

°  Provides  information  on  how  to  generate  a  risk 
management  plan  based  on  the  results  of  the 
evaluation 


Created  a  Risk-Based  Assessment  Tool  (RBAT) 
°  Comprehensive  questionnaires 
°  Scoring/Weighting 
°  Tailoring  (adding  /  deleting) 

°  Summary  roll-up 


Screen  Shot  of  Summary  Sheet  of  RBAT 


Pre-Award  Subcontractor  Assessment 


\l.2  1 2/2  u, ■'28)04  This  spread  shea 


A 


Hide  Hints 


\J)\  Intro  I 


Subcontractor: 

Company: 

Address: 


Advance  to  next  worksheet  _  -  Help  for  this  worksheet  —  Jump  to  Introduction 

- Hide  these  annoying  boxes  ument  Date: - 


inter  the  Contact  Data  relevant  to 

his  particular  Subcontractor  &  _ 

assessment  in  these  boxes 


Slate: 

Zip: 


Country: 

Province: 

Name: 

Title: 

Email: 


_  Itact  (POC): 


Phone: 

Fax: 

Assessment  Team: 

Assessor  POC: 

Title: 

Email: 

Assessor  Names: 


Data  from  here  on  down  are  automatically 
calculated  on  the  Risk  Area  worksheets. 
Seethe  Introduction  worksheet  for 


Phone: 

Fax: 


Notes: 


Overall  Results 

0 

0 

Weighted: 

Click  to  jump  to  that  Risk  Area  Worksheet 

Penetration 

Weight 

Factor 

Weighted 

Score 

x  to 

include 

Risk 

Area 

Topic 

Score 

Actual 

Max 

Possible 

X 

1 

Reouirements  Manaoement 

0 

0 

X 

2 

Project  Plannina 

0 

0 

X 

_ 

Project  Monitor  a mt-T nntrol 

0 

© 

3 

Plaro  an  Vr  in  1 

hie  rnliimn  tn 

1 Mi  Click  on  the  Risk  Area  number  or 

J- 'i-jl  Topic  to  jump  to  that  worksheet 

0 

0 

include  this  Risk  Area  in  the 
Assessment:  make  it  blank  to  hide 
that  worksheet 

0 

0 

i 

:t  QA - 

0 

aement 

A  '  U'  '  |  F 

+  r* i  r  +  r* ■  o  -a  h  D  i  1' 

loDment 

rTLoeiyii  a.  yhtiljiii.  i  cillui  lu  ccil.ii  ilioh. 

Area  which  can  increase/decrease 
it's  significance  in  the  overall  results 

X 

5 

r\t:uuu ueve 

X 

a 

Technical  Solution 

X 

10 

Product  Integration 

Screen  Shot  of  Risk  Sheet  of  RBAT 


■;<=!<  niiMniimgftETiBPT 

Description: 


The  purpose  of  Requirements  Management  (REQM)  is  to  Navi 
1  Developing  an  understanding  of  the  requirements  fn< 

2.  Obtaining  commitment  from  project  participants,  Maiifi 

3.  Managing  changes  to  requirements  -  as  they  evolve,  > 

4.  Maintaining  traceability  between  requirements  and  pla 

5.  Identifying  inconsistencies  between  those  requirements  arid  the  project's  plans  and  work  products 


Hide  Hints 


Main  I 


11M3U9. 


gatson 

Jump  to  previous  Worksheet 
Jump  to  the  Summary  Worksheet 
Advance  to  next  Worksheet 


leCKKltB  u 


.77777^ 


View  Risk  Area  — j  t - \ 

pr  ±-  jet  )i r""" 

questions  Help  for  this  worksheet 


Spell  Check 

Evaluator 

Notes 


Question 


1.0 


projects. 


Listen  For 

What  to  listen  form  the 
Describe  how  Subcontractor's  response  to  the 
question.  Not  included  in  the  Pre- 
Assessment  workbook. 


Listen  For 


Change  control,  meeting  with 
"customer"  to  understand  intent, 
templates  for  inclusion  of 
l^ri  mary/supplementary 

Elements  data,  relationships 
established  and  maintained 


Questions 

:  What  to  ask  the  Subcontractor.  High-level 
:  probing  questions  end  in  .0  with  a  yellow 
:  background.  Subsequent  questions  that  are 
r  right  aligned  are  for  follow-up.  Only  Probing 
|  questions  will  be  included  in  the  Pre- 
:  Assessment  workbook. 


Objective  Evidence 
What  to  lor  k  for  from  the  Subcontractor 
when  asse  ssing  how  well  they  answer  the 
question.  C  ptionally  included  in  the  Pre- 
Assessment  workbook. 


Describe  your  change  control  process. 


i  What  tools  are  utilized  to  support  the  requirements 
\  management  activities  during  the  engineering 
;  development  life  cycle? 


;  At  what  point  are  requirements  placed  under  formal 


Internal  reviews  wiliyM 


documents  &  other  work  products 


A  process  or  procedure,  a  "formal" 
review,  placing  under  configuration 
management  control,  existence  of 
a  Change  Control  Board  or 
eq 


Objective  Evidence 


1.  Meeting  minutes  from  working 
group  or  technical  exchange 
meetings 

2.  Requirements  management 
policy 

3.  Tools  used,  sample  templates 
reports 


Evaluator  Notes 


Sign  off  sheets  from  meetings 
with  list  of  "organizations" 
represented  -  not  just  names 


K 


Completion  column 

Y  To  signify  when  a  question  is  completed 

V  No  result  is  attached  to  this  type  of  item 
used  to  introduce  a  high-level  probing 
question 

blank  Question  has  not  been  resolved. 

Use  dropdown  box  to  choose 


(need  an  Qrg.  chart  to  traci  Results 


.  Checklists  that  include 
documents  to  be  reviewed 
changes  from  requirement 
changes 

.  Group  reviews  /  minutes 


1.  Procedure  or  documented 
process 


Rotes 

Room  to  detail  their  response.  Only  the 
first  1000  characters  will  show  when  you 
click  on  the  cell.  32000  max 


Final  Score  given  for  question.  Use  dropdown  box  ?  J" 
to  choose:  S. 

5  Fully  compliant 
3  Partially  compliant 
0  Deficient 
n/e’  Not  Evaluated 

Lefthand  side  is  a  Probing  Question,  right-hand  side 
is  a  follow-up.  An  underlined  Probing  value  means 


2.  Signoff  sheet  form  (request  it  is  the  average  of  all  related  follow-up  questions, 
change) 


Change  Control  Board  minite  \ 
ports 


rements  baseline  report 


What  risks  or  issues  do  acquirers  see  from 
working  with  small  suppliers? 


STRENGTH  THKOLGII INDUSTRY  &  TEOLNOLAXiY 


Risks  /  Issues 

Acquirer  Mitigation  Approach? 

•Processes:  completeness,  integration? 
•Product  reqts,  interfaces,  integration? 
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How  do  you  approach  integrating  prime/supplier 
processes  and  products? 


STRENGTH  THKOLGII INDUSTRY  &  TEOLNOLAXiY 


Process/product  integration  techniques 

•Common  processes/tools?  Integrated  processes/tools?  Separate  processes/tools? 
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What  recommendations  do  you  have  for  small 
businesses  interested  in  adopting  CMMI? 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Recommendations 

•Look  at  your  teams  -  PSP,  TSP.  Within  2  years  planning  processes  will  be  in  place.  Do 
gap  analyses  to  improve.  Achieve  ML3  in  36  months,  Level  5  using  PSP/TSP. 

•Use  the  company’s  culture  and  terminology  to  communicate  in  their  terms  -  not  CMMI. 

•Appraisals  used  to  be  discovery  oriented  -  appraisal  did  the  evaluation  and  mapping  to 
CMMI.  Now,  the  emphasis  is  on  preparing  artifacts  and  evidence,  with  little  value  to  the 
business. 

•Top  management  buy-in  -  important  in  companies  of  all  sizes. 

•Language  used  in  CMMI  can  be  difficult  to  relate  to  -  more  in  the  context  of  a  large 
defense  contractor.  Translation  to  a  small  company  can  be  very  difficult. 

•Treat  process  improvement  like  a  project  -  project  plans,  etc. 


CMMI®  for  Small  Business 
NDIA  CMMI  Working  Group 
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For  More  Information 


STRENGTH  T1IKOIGII INDUSTRY  &  TECHNOLOGY 


NDIA  CM  Ml  Working  Group 

http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/Pages/CMMI  Working  Group.aspx 


Jim  Armstrong 

Stevens  Institute 

Geoff  Draper 

Harris  Corporation 

Wendell  Mullison 

General  Dynamics, 
Land  Systems 


Steve  Austin 

Lockheed  Martin 

Jeffrey  L.  Dutton 

Jacobs  Technology 

Rick  Welch 

Northrop  Grumman 


Dan  Blazer 

SAIC 

Nancy  Fleischer 

Raytheon  Company 


Michael  Campo 

Raytheon  Company 

Kathy  Smith 

Hewlett  Packard  EDS 


With  special  appreciation  to  our  panelists: 


Lynn  Penn,  Lockheed  Martin 
Rosalind  Singh,  Raytheon  Company 


Rick  Welch,  Northrop  Grumman 

Girish  Seshagiri,  AIS 


CMMI®  for  Small  Business 
NDIA  CMMI  Working  Group 
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Risk  Based  Subcontractor 

Assessment 

Lockheed  Martin  Corporation 


Risk  Based  Subcontractor 
Assessment  Topics 

:  Issues 

:  Identification  of  Risk  Areas 
:  Developed  Resources 
:  Initial  and  On-going  Evaluations 
:  Risk  Areas  for  Evaluation 
:  Risk  Area  Questionnaire 
:  Risk  Report 
:  Summary 


Issues  -  Impacts  to  Mission  Success 


►  Analysis  has  shown  that  engineering  subcontractors  may 
impact  Lockheed  Martin  Company  program  success 

°  Subcontract  content  and  design  responsibility  on  programs 
continues  to  grow 

Lack  of  diligence  in  subcontract  management  can  be  a  primary 
cause  of  program  failure 

►  Issues  become  more  obvious  when  the  subcontractor  is: 

°  A  major  program  contributor  (greater  than  1 0%  of  the  total 
contract  value) 

°  A  Sole  Source  supplier 
°  On  the  program  critical  path 
Considered  a  medium  to  high  risk  to  program  performance 
°  New  to  Lockheed  Martin  with  no  performance  history 

A  past  supplier  to  Lockheed  Martin  with  problematic  performance 
history 


Identification  of  Risk  Areas 

►  Multiple  studies  conducted  to  identify  key 
risk  areas 

°  Multiple  recurring  problem  themes 
°  Results  were  consistent  with  the  Software 
Engineering  Institute’s  CMMI  for  Development  model 

►  Adopted  the  framework  of  CMMI-DEV  to 
focus  on  the  likelihood  of  a  subcontractor’s 


success 


Developed  Resources 


► 


► 


Created  Risk-Based  Subcontractor  Assessment 
Guidebook: 

°  Includes  initial  and  on-going  evaluations 

°  Defines  assessment  team  roles  and  responsibilities 

°  Explains  how  and  who  to  interview  during  a 
subcontractor  evaluation 

°  Provides  information  on  how  to  generate  a  risk 
management  plan  based  on  the  results  of  the 
evaluation 


Created  a  Risk-Based  Assessment  Tool  (RBAT) 
°  Comprehensive  questionnaires 
°  Scoring/Weighting 
°  Tailoring  (adding  /  deleting) 

°  Summary  roll-up 


Screen  Shot  of  Summary  Sheet  of  RBAT 
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Product  Integration 

Screen  Shot  of  Risk  Sheet  of  RBAT 
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The  purpose  of  Requirements  Management  (REQM)  is  to  Navi 
1  Developing  an  understanding  of  the  requirements  fn< 
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3.  Managing  changes  to  requirements  -  as  they  evolve,  > 

4.  Maintaining  traceability  between  requirements  and  pla 

5.  Identifying  inconsistencies  between  those  requirements  arid  the  project's  plans  and  work  products 
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questions  Help  for  this  worksheet 


Spell  Check 

Evaluator 

Notes 


Question 


1.0 


projects. 


Listen  For 

What  to  listen  form  the 
Describe  how  Subcontractor's  response  to  the 
question.  Not  included  in  the  Pre- 
Assessment  workbook. 


Listen  For 


Change  control,  meeting  with 
"customer"  to  understand  intent, 
templates  for  inclusion  of 
l^ri  mary/supplementary 

Elements  data,  relationships 
established  and  maintained 


Questions 

:  What  to  ask  the  Subcontractor.  High-level 
:  probing  questions  end  in  .0  with  a  yellow 
:  background.  Subsequent  questions  that  are 
r  right  aligned  are  for  follow-up.  Only  Probing 
|  questions  will  be  included  in  the  Pre- 
:  Assessment  workbook. 


Objective  Evidence 
What  to  lor  k  for  from  the  Subcontractor 
when  asse  ssing  how  well  they  answer  the 
question.  C  ptionally  included  in  the  Pre- 
Assessment  workbook. 


Describe  your  change  control  process. 


i  What  tools  are  utilized  to  support  the  requirements 
\  management  activities  during  the  engineering 
;  development  life  cycle? 


;  At  what  point  are  requirements  placed  under  formal 


Internal  reviews  wiliyM 


documents  &  other  work  products 


A  process  or  procedure,  a  "formal" 
review,  placing  under  configuration 
management  control,  existence  of 
a  Change  Control  Board  or 
eq 


Objective  Evidence 


1.  Meeting  minutes  from  working 
group  or  technical  exchange 
meetings 

2.  Requirements  management 
policy 

3.  Tools  used,  sample  templates 
reports 


Evaluator  Notes 


Sign  off  sheets  from  meetings 
with  list  of  "organizations" 
represented  -  not  just  names 


K 


Completion  column 

Y  To  signify  when  a  question  is  completed 

V  No  result  is  attached  to  this  type  of  item 
used  to  introduce  a  high-level  probing 
question 

blank  Question  has  not  been  resolved. 

Use  dropdown  box  to  choose 


(need  an  Qrg.  chart  to  traci  Results 


.  Checklists  that  include 
documents  to  be  reviewed 
changes  from  requirement 
changes 

.  Group  reviews  /  minutes 


1.  Procedure  or  documented 
process 


Rotes 

Room  to  detail  their  response.  Only  the 
first  1000  characters  will  show  when  you 
click  on  the  cell.  32000  max 


Final  Score  given  for  question.  Use  dropdown  box  ?  J" 
to  choose:  S. 

5  Fully  compliant 
3  Partially  compliant 
0  Deficient 
n/e’  Not  Evaluated 

Lefthand  side  is  a  Probing  Question,  right-hand  side 
is  a  follow-up.  An  underlined  Probing  value  means 


2.  Signoff  sheet  form  (request  it  is  the  average  of  all  related  follow-up  questions, 
change) 


Change  Control  Board  minite  \ 
ports 


rements  baseline  report 


Initial  and  On-Going  Evaluations 


►  Differences  in  two  types  of  Evaluations 


Initial 

On-Going 

When 

Prior  to  Contract  Award 

Anytime  after  Contract  Award: 
Regular  intervals  or  on  demand 

Duration 

1-2  days 

1  day 

Emphasis 

Generalized  Fact  Finding 

Project  implementation  and  risk 

Quantitative 

Results 

Important 

Yes  -  for  multiple 
Subcontractor  Evaluation 

No  -  for  sole  source 

No 

Scope  of  Review 

Subcontractor 

Subcontractor,  Lockheed  Martin 
Prime,  Customer 

Risk  Areas 

1  ^ — - 

Potentially  all 

Can  be  more  focused, 
depending  on  the  lifecycle 
phase 

Risk  Area  -  Starter  Set 


17  key  Risk  Areas,  but  what  constitutes  a  good  “starter 
set”? 

::  Requirements  Management 
::  Risk  Management 
::  Verification 
::  Validation 

::  Measurement  and  Analysis 

::  ...and  perhaps  Supplier  Agreement  Management,  if  next 
tier  suppliers  exist 


Risk  Area  Questionnaire 

Probing  a  Risk  Area  -  Best  Practices 

::  Keep  it  simple 

::  Limited  number  of  good  questions 
::  Backup  questions  (for  added  clarity) 

::  Review  objective  evidence  if  available 
::  Be  mindful  of  expected  responses 
::  Look  for  integration  themes  (threading) 

::  Tailor/add/delete/prioritize  questions  for  effectiveness 
::  Note  risks  as  they  are  uncovered 

::  Keep  Subcontractor  grading  feedback  to  a  minimum  during 
evaluation 

::  Limit  evaluation  duration  (within  a  day  in  most  cases) 

::  Generate  a  risk  report  and  execute  to  it 


Risk  Reports 

ss  No  Subcontractor  is  a  perfect  match 

ss  Using  a  standard  evaluation  strategy  and  querying 
multiple  potential  risk  areas  should 

ss  Provide  a  more  accurate  portrayal  of  Subcontractor 
capability  earlier  in  the  lifecycle 

ss  Enable  risk  management  kickoff,  when  necessary 

ss  Ultimately  strengthen  the  LMC/Subcontractor  relationship 


Summary 

►  Guidebook  and  Tool  provide  our  LM 
companies  with  information  that  enables: 

°  More  confident  source  selection 

°  On-going  review  /  risk  management  during  a 
program’s  development  cycle 

°  More  sharing  of  subcontractor  evaluation  data  across 
the  corporation 

►  Process  and  tool  can  be  used  with  any  size 
Subcontractor 


Completing  an  assessment  AND  proactive  risk 
management  compose  the  solution  framework 


Process  Alignment 

From  Aligned  to  Aligner 


Ilene  Pinsker 

Sr.  Systems  Engineer 

Rockwell  Collins,  Inc. 

Simulation  and  Training  Solutions 
November,  2010 


Rockwell 

Collins 


Rockwell 
Collins 

Agenda 

•  Background  Information 

•  Simulation  and  Training  Systems  (STS)  Alignment  to  Rockwell 
Collins  (RC) 

•  Visual  Systems  (VS)  Alignment  to  STS 

•  Observations  and  Lessons  Learned 

•  Ongoing  Activities 
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Rockwell  Collins,  Inc.  (RCI) 


•  International  provider  of 
communication  and 
aviation  electronics 
solutions 

•  Mature  processes 

•  CMMI  Level  3 

•  Developing  plans  for  CMMI 
future 
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NLX 


•  Small,  privately  owned  company 

•  Built  military  and  commercial  full  flight  simulators 

•  Engineering  facilities  in  Sterling  VA,  Binghamton  NY  and 
Huntsville  AL 

•  Immature  processes 

•  ISO  certified 

•  Developing  plans  for  CMMI  Level  3 

•  Acquired  by  Rockwell  Collins  in  December,  2003 

-  NLX  became  Rockwell  Collins  Simulation  and  Training  Solutions 
(STS) 


Rockwell 
Collins 

Evans  and  Sutherland  (E&S) 

•  Small  company  which  created  visual  image  generators 

•  Provided  many  of  the  image  generators  used  in  the  simulators 
built  by  STS 

•  Engineering  facilities  in  Salt  Lake  City  UT,  Orlando  FL  and 
Horsham,  England 

•  Immature  processes 

•  ISO  certified 

•  No  plans  for  CMMI 

•  Acquired  by  Rockwell  Collins  Simulation  and  Training  Solutions 
in  May,  2006 

-  Evans  and  Sutherland  became  STS  Visual  Systems  (VS) 
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Rockwell 
Collins 

Alignment  Objectives 

•  Integrate  STS  into  the  RC  Quality  Management  System 

•  Transition  to  RC  Enterprise  Tools 

•  Retain  STS  Best  Practices 

•  Maintain  ISO  9001:2000  Certification  during  transition  and 
after  alignment 

•  Achieve  CMMI  Level  3 
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Timeline 


May-05 

First  STS  Project 
Uses  RC  Process 


Dec-03 

RC  Acquires  NLX 


0 


0 

- 1 — 

Jan-04 


0 


Jan-05 


Jan-03 

NLX  Initiates  CMMI  Pursuit 


May-06 

STS  Acquires  E&S 


C> 

1 - 


♦ 


Jan-06 


Jan-07 


Jan-08 


Mar-08 

STS  Achieves  CMMI  Level  3 
(Excluded  VS) 
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RC  Contrast  To  STS 


•  Rockwell  Collins  product 

-  Transitions  to  factory  production 

-  Requires  FAA  or  equivalent  certifications,  such  as  D0178B 

•  STS  product 

-  Each  simulator  contract  is  for  a  custom-built  device  which  doesn't 
require  large  scale  production 

-  Simulators  are  not  flight  worthy  systems,  so  they  don't  require  FAA 
certifications 
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Rockwell 

Collins 


STS  and  CMMI 

•  Although  ISO  certified,  the  CMMI  requirements  brought  many 
new  challenges  for  STS 

-  No  one  at  STS  knew  the  CMMI 

•  2  people  were  sent  to  the  SEI  for  the  Introduction  to  the  CMMI  training 

-  STS  processes  merely  provided  a  starting  point  for  CMMI 

•  Too  high  level 

•  Some  process  areas  not  included 

-  Where  do  we  begin? 
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Process  Alignment 


•  Rockwell  Collins  processes  were  CMMI  compliant! 

•  Can  the  same  organizational  standard  process  be  used  for 
simulators? 

•  What  about  the  legacy  STS  processes? 

•  The  vetting  process  began  with  much  push-back  from  the 
practitioners  that  did  not  welcome  change 

-  "The  process  is  oriented  to  a  production  environment" 

-  "We  don't  build  flight  worthy  systems  and  don't  have  the  same 
safety  requirements" 

-  "The  people  in  Cedar  Rapids  don't  understand  our  unique  product 
needs" 

-  "Our  process  is  working,  why  do  we  need  to  change  it?" 

-  "Our  customers  are  happy,  why  change?" 
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Process  Alignment 


•  Representatives  from  Rockwell  Collins  provided  mentoring  on  the  RC 
Design  and  Development  processes 

•  STS  representative  added  to  the  RC  Government  Systems 
Organizational  Process  Group  (OPG) 

•  STS  Team 

-  Lead  by  Engineering  Processes  and  Tools  and  Software  Quality 

-  Functional  area  representatives 

•  Quality  Engineering 

•  Engineering  -  Systems,  Hardware  and  Software 

•  Operations 

•  Manufacturing 

•  Project  Management 

•  Each  STS  process  was  compared  to  the  RC  processes 

-  Is  there  an  existing  RC  process? 

•  Can  RC/STS  use  it  as  is? 

•  If  not,  can  it  be  modified  to  include  unique  product  requirements? 

•  If  not,  convert  the  legacy  STS  process  to  RC  format 

-  Due  to  tailorability  of  RC  process,  few  processes  were  converted 

•  New  programs  were  required  to  follow  the  resulting  process 

-  RC  provided  training  as  well  as  mentoring 

-  Institutionalization  was  gradually  achieved 
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Initial  Findings 


RC  Process 

STS  Legacy 
Process 

Result 

Management 

Entire  life  cycle  (Business 
opportunity  identification 
through  program  completion) 

Starts  at  contract 
award 

Replace  legacy  process 
with  RC  process 

Engineering 

•  RC  technical  process  includes 
areas  that  are  not  applicable  to 
STS  (i.e.  ASIC,  manufacturing 
transition) 

•Well  defined  tailoring  tool 

Hardware  and  software 
process  can  be  directly 
mapped  to  RC  process 

New  programs  to  use 

RC  process  and  tailor 
out  areas  that  are  not 
applicable 

Quality 

•Software  Quality  Engineering 
(SQE) 

•  Design  Quality  Engineering 
(DQE) 

•Quality  Inspectors 

•  Newly  Software 

Quality  position 

RC  to  mentor 
development  of  SQE 
and  DQE  positions  at 

STS 

CM 

Process  includes  configuration 
management  and  control 

•CM  process  can  be 
mapped  to  RC 

•Toolset  is  different 

New  programs  to  use 

RC  CM  process  and 
tools 
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Rockwell 
Collins 

Tool  Alignment 

•  Engineering  Tools 

-  Process  does  not  require  use  of  specific  tools,  but  many  are  highly 
recommended 

-  Each  RC  standard  tool  was  evaluated  individually,  this  often  included  pilots 

-  STS  representative  on  the  RC  Tool  Disposition  Board 


RC  Standard 

STS 

Result 

Design  and 

Development 

Process 

RC  developed  tailoring 
tool 

No  formal  tool 

Use  RC  tool 

Requirements 

Management 

DOORS 

Excel  Spreadsheet 

DOORS  is  recommended,  but 
decision  is  program  dependent 

Configuration 

Management 

Subversion 

Razor 

New  programs  use  Subversion 
Legacy  programs  continue  with 
existing  tool 

Data  Repository 

TeamSpace  -  RC 
application  of  Microsoft 
Windows  SharePoint 
Services 

Shared  Drives 

Electronic 

Database 
Management 
System  (EDMS) 

STS  programs  continue  to  use 

EDMS 

Peer  Reviews 

RC  developed  tool 

EDMS 

Continue  to  use  EDMS 
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Roll  Out 


•  Communication 

-  Briefings  to  senior  management  and  program  management 

-  Quality  oversight  during  program  planning 

-  Informative  e-mails 

•  Training 

•  Pilot  project 

-  New  full  flight  simulator 

-  Required  to  achieve  CMMI  Level  3  within  36  months  of  contract 

•  Highly  motivated  to  use  the  RC  CMMI  compliant  processes 

-  Transition  team  mentoring 

•  Trained  project  team  on  new  process  and  tools 

•  Guided  project  leaders  through  the  tailoring  process 

•  Worked  with  team  to  assure  all  process  required  artifacts  were 
developed 

•  Quality  audit  checklists  developed  to  assure  compliance  to  the 
project's  tailored  process 
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Rockwell 

Collins 


CMMI  Level  3  Achievement 

•  Shortly  after  roll  out,  CMMI  activities  began 

•  The  need  for  CMMI  provided  impetus  for  institutionalization 

•  Cedar  Rapids  team  performed  CMMI  Internal  Evaluations 

-  July,  2006  and  October,  2007 

•  Class  C  Appraisal  December,  2007 

•  CMMI  Level  3  achieved  March,  2008 


May-05 

First  STS  Project 
Uses  RC  Process 


Jul-06 

Internal 

Evaluation 


Sep-07 

Internal 

Evaluation 


<> 


♦ 


♦  <> 


Jan-06 


Jan-07 


Jan-08 


Dec-07 
Class  C 
Appraisal 


Mar-08 

STS  Achieves  CMMI  Level  3 
(Excluded  VS) 
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Rockwell 

Collins 

Visual  Systems 

•  The  transition  from  STS  to  RC  processes  was  well  underway 
when  STS  Visual  Systems  (VS)  entered  the  picture 

•  They  needed  to  align  with  the  RC  Design  and  Development 
processes  to  eventually  achieve  CMMI  Level  3 

•  The  VS  product  transitions  to  the  factory 

•  VS  programs  customize  the  basic  product  for  specific  purposes 
with  no  factory  production 

•  The  role  of  the  transition  team  was  now  reversed 

•  Where  do  we  begin? 
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Been  There,  Done  That 


•  Recent  alignment  activities  gave  transition  team  good 
perspective 

-  Desire  to  retain  existing  process 

-  Need  for  buy-in  from  practitioners 

•  "The  STS  process  is  not  oriented  to  a  production  environment" 

•  "You  don't  understand  our  unique  product  needs" 

•  "Our  process  is  working,  why  do  we  need  to  change  it?" 

•  "Our  customers  are  happy,  why  change?" 

•  "We  have  many  very  short  duration  programs" 

-  Analyze  current  processes  and  tools 

•  Identify  process  gaps 

•  Identify  and  retain  best  practices 


17 


Process  Alignment 

•  Quality  audits  to  identify  process  gaps 

-  Used  the  checklists  which  were  developed  to  audit  STS  projects 

-  Assigned  actions  to  close  gaps 

•  Start  following  process 

•  Tailor  process  for  unique  VS  needs 

•  Representatives  from  STS  provided  mentoring  on  the  RC  Design  and 
Development  processes  with  support  from  RC 

•  RC  processes  were  evaluated  for  applicability  to  VS 

-  Can  the  process  be  used  as  is? 

-  If  not,  can  it  be  modified  to  include  unique  VS  requirements? 

-  If  not,  create  a  new  process 

•  VS  Team 

-  Lead  by  VS  Quality  and  STS  Engineering  Process  and  Tools 

-  Functional  area  representatives 

•  Quality  Engineering 

•  Engineering  -  Systems,  Hardware  and  Software 

•  Project  Management 

•  Product  Management 

-  New  programs  were  required  to  follow  the  resulting  process 

-  STS  and  RC  provided  training  as  well  as  mentoring 

-  Institutionalization  is  progressing 
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Initial  Findings 


RC/STS  Process 

VS  Legacy 

Process 

Result 

Management 

Entire  life  cycle  (Business 
opportunity  identification 
through  program  completion) 

Starts  at  contract 
award 

VS  needs  to  replace 
legacy  process  with  RC 
process 

Engineering 

•  RC  technical  process 
•Well  defined  tailoring  tool 

Hardware,  software 
and  database 
processes  can  be 
directly  mapped  to  RC 
process 

New  programs  can  use 
RC  process  and  tailor 
out  areas  that  are  not 
applicable 

Quality 

•Software  Quality  Engineering 
(SQE) 

•  Design  Quality  Engineering 
(DQE) 

Quality  Inspectors  and 
Engineers  -  not  same 
as  the  SQE  and  DQE 
role 

STS  to  mentor 
development  of  SQE 
and  DQE  positions  at 

VS 

CM 

Process  includes  configuration 
management  and  control 

•CM  process  can  be 
mapped  to  RC 

•Toolset  is  different 

New  programs  will  use 
RC  CM  tools 
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Rockwell 
Collins 

Tool  Alignment 

•  Engineering  Tools 

-  Each  RC  and  STS  standard  tool  was  evaluated  individually,  this  often  included 
pilots 


STS 

VS 

Result 

Design  and 

Development 

Process 

RC  developed  tailoring 
tool 

No  formal  tool 

Use  RC  tool 

Requirements 

Management 

DOORS  or  Excel 
Spreadsheet 

No  formal  tool 

Transition  to  DOORS 

Configuration 

Management 

Subversion 

ClearCase 

•  New  programs  use 

Subversion 

•  Legacy  programs  continue 
with  existing  tool 

Data  Repository 

Electronic  Database 
Management  System 
(EDMS) 

•TeamSpace  -  RC 
application  of 

Microsoft  Windows 
SharePoint  Services 

•Shared  Drives 

New  programs  use  EDMS 

Peer  Reviews 

EDMS 

No  formal  tool 

EDMS 
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Roll  Out 


•  Communication 

-  Briefings  to  senior  management  and  program  management 

-  Quality  oversight  during  program  planning 

-  Informative  e-mails 

•  Training 

•  Programs  that  supplied  the  image  generator  for  STS  programs 

-  Already  following  parts  of  the  process  as  part  of  the  STS  team 

•  New  product  development 

•  Include  in  next  CMMI  appraisal 

•  Transition  team  mentoring 

-  Process  and  tool  training 

-  Guide  project  leaders  through  the  tailoring  process 

-  Work  with  team  to  assure  all  process  required  artifacts  were 
developed 

•  Quality  audits  to  project's  tailored  process 
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Observations  After  Being  On  Both  Sides 


•  Practitioner  buy-in  is  critical  to  success 

-  Listen  to  them 

-  Acknowledge  their  concerns 

-  Include  them  in  the  process 

-  Assure  they  understand  the  reason  for  change 

•  Leverage  prior  integration  experience 

•  Audit  to  the  acquirer's  process  to  identify  gaps 

-  Alignment  may  be  closer  than  you  think 

•  Don't  mandate  process  change 

-  Communicate  the  business  case  for  the  change 

-  "Because  I  said  so"  will  lead  to  a  process  that  no  one  follows 

•  The  basic  design  and  development  process  is  the  same  in  spite 
of  varied  end  products 
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Lessons  Learned 


•  Learn  and  incorporate  the  best  practices  of  the  legacy  process 

-  Don't  assume  the  "acquirer"  does  it  better.  The  basic  Design  and 
Development  process  is  the  same  for  all  products 

•  Tailoring  allows  for  unique  business  needs 

•  Mentoring  is  critical  to  achieve  understanding 

-  Include  a  process  advocate  at  each  facility 

•  Distance  is  a  challenge 

-  Involve  all  business  locations  to  assure  buy-in 

-  Acquirer  alignment  team  members  should  be  at  acquirer  facility  during  the 
transition 

•  Assign  specific  actions,  set  deadlines  and  cadence 

-  Run  the  transition  as  a  program 

•  Transition  as  soon  after  acquisition  as  possible 

•  Select  a  pilot  program 

•  Audit  to  the  new  process  early  in  the  project 

•  The  practitioners  will  resist  change 

•  Communication  is  critical  to  avoid  duplication  of  efforts 


23 


Ongoing  Activities  to  Maintain  Alignment 


•  Mentoring 

-  Enterprise  level  project  start-up  teams 

•  Process  tailoring 

•  Project  planning 

•  Design  and  Development  Governance  Council 

-  Approves  new  and  updated  processes 

-  Representation  from  multiple  RC  Business  Units  and  locations 

•  Assures  unique  product  needs  are  considered 

•  Internal  Audits 

-  Quality  audits  use  the  same  checklists  and  criteria  to  assure  all 
locations  are  adhering  to  the  process 
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Rockwell 

Collins 
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Rockwell 

Collins 
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Rockwell 

Collins 


Backup 

•  NLX  QMS 

-  125  Documents 

•  108  Procedures 

•  17  Work  Instructions 

-  83  Obsolete 

-  24  RC 

•  E&S  QMS 

-  Approximately  125  Documents 

-  3  converted  to  RC 


27 


VS  Quality  Audit  Result 


Process 

Score 

Set  Direction 

0.75 

Risks  and  Opportunities 

0.65 

Plan  Project 

0.28 

Execute  Project 

0.73 

Evaluate  Project 

0.55 

Complete  Project 

0.75 

Capture  Originating  Requirements 

0.83 

Define  Operational  Concepts 

0.25 

Define  Requirements 

0.85 

Design  Solution 

0.90 

Implement  Solution 

0.94 

Integrate  Solution 

0.83 

Develop  Acceptance  Procedures 

1.00 

Develop  Validation  Cases  and  Procedures 

1.00 

Verify  Solution 

0.75 

Support  Solution 

0.94 

Legend 

0.9  -  1.00 

0.5  -  0.89 

0.0  -  0.49 

Air 

Land 

Sea 

Space 
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■  Raytheon  Network  Centric  Systems  Composition 

■  The  Challenge 

■  The  Solution 

■  Corrective  Action  Aging 

■  Overdue  Corrective  Action 

■  External  CARs 

■  External  Audits 

■  Internal  Audits 

■  Supplier  Assessments 
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The  Challenge 


Raytheon 

Network  Centric  Systems 


■  Process  Evaluations  conducted  at  the  project  level 

-  No  organizational  picture  of  process  performance 

■  Process  Performance  tracked  at  the  regional  level 

-  Multiple  tools  used 

•  Excel 

•  Oracle 

.  CAWeb 
.  SQIDS 

•  Filemaker  Pro 

•  ClearQuest 

•  iTracker 

■  No  organizational  visibility  into  issues  identified  or  processes  evaluated 

■  Process  Evaluations  were  engineering  centric 

-  Separate  from  ISO  audit  function 

■  Issue  identification  variability 

■  Increase  in  DCMA  audit  and  escalation  activity 
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The  Solution 


Raytheon 

Network  Centric  Systems 


■  Adopted  the  Raytheon  Corrective  Action  System  (RCAS) 

-  Corrective  Action,  Audits,  and  Supplier  Modules 

■  Nationalized  the  Process  Evaluation  Process 

■  Extended  the  Process  to  all  functions 

■  Set  goals  for 

-  On  time  process  evaluation  conduct  vs  plan 

-  On  time  corrective  action  response,  implementation  and  closure 

-  On  time  customer  corrective  action  closure 

-  Reduction  in  overdue  corrective  actions 

■  Developed  and  delivered  training  to  all  regions  regarding  management 
expectations  regarding  corrective  action 
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What  we  did 


Raytheon 

Network  Centric  Systems 


■  Established  Regional  Audit  Program 

■  Expanded  audits  to  functions  outside  of  engineering 

■  Expanded  to  include  ISO,  CMMI,  Corporate  and  NCS  P&Ps 

■  Audits  are  now  risk  based  to  determine  frequency 

■  Review  with  Program  Leadership  and  Engineering 

■  Developed  a  series  of  measures  and  performance  baselines 

-  Reported  monthly  to  leadership 

-  Baseline  reduction  activities  are  continuously  brainstormed 

-  Embarrass  the  local  leaders 
-Regional  competition 
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Establish  Process  Performance 
Measures 


Raytheon 

Network  Centric  Systems 


■  Regional  Process  Evaluations 
-Scheduled  vs.  completed 
-Issues  per  Evaluation 

■  CAR  Cycle  time 

■  Customer  CAR  Volume  (level  2  and  above) 

■  Supplier  Process  and  Product  Evaluations 

■  Supplier  Corrective  Action  Cycle  time 
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Establish  Quality  Objectives 


Raytheon 

Network  Centric  Systems 


■  On  time  process  evaluations 

■  Timely  Corrective  Action  closure 

■  Reduction  in  cycle  time 

■  Timely  Supplier  CA  closure 

■  Measurements  are  taken  at  the  regional  and  organizational 
level 


Page  9 


Establish  Process  Performance 
Baselines 


Raytheon 

Network  Centric  Systems 


■  Process  performance  baselines  are  defined  as  each 
measures  distribution  and  range  of  results 

■  The  goal  is  to  understand  the  variation  and  improve  the 
process 

-  Process  evaluations  on  schedule 

-  Reduction  in  CAR  cycle  time 
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Corrective  Action  Aging 


Raytheon 

Network  Centric  Systems 


Breakdown  of  CA  Aging 


Baseline  is  less  than  25%  of  open  corrective  actions  are  over  60  days 
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Corrective  Action  Aging  Raytheon 

NCS  August  Fiscal  Month  End  2010  Network  Centric  Systems 


NCS  Corrective  Action 
Aging 


Meeting  goal  -  continuous  improvement  in  reducing  cycle  time 


Over  Due  Corrective  Actions 
August  2010 


Raytheon 

Network  Centric  Systems 


Sept  Oct  Nov  Dec  Jan  Feb  Mar  Apr  May  June  July  Aug 


Business  Level  Trend 


Site 

Overdue 

CAs 

% 

Overdue 

Total  CAs 

Dallas- 

Lemmon 

0 

0% 

11 

Falls  Church 

0 

0% 

4 

Ft.  Wayne 

1 

4% 

23 

Fullerton 

(RFO) 

0 

0% 

10 

Fullerton 

(TRS) 

0 

0% 

5 

Goleta 

0 

0% 

25 

Largo 

0 

0% 

68 

Marlborough 

2 

11% 

18 

Orlando 

0% 

r 

0 

Bardmoor 

0 

0% 

25 

Texas 

3 

9% 

35 

Tows  on 

0 

0% 

3 

Waterloo 

0 

0% 

13 

RSL 

Totals 

6 

3% 

240 

Breakdown  by  Site 


Exceeding  the  goal  (2nd  month) 
Oversight  continues  to  improve  delinquent  CAs 


CUSTOMER  CARS 
August  2010 _ 


Raytheon 

Network  Centric  Systems 


All  CARs  are  in  work  and  being  reviewed  weekly  with  Mission  Assurance  Leadership 


Process  Evaluation 
Performance  Baseline 


Raytheon 

Network  Centric  Systems 


30%  increase 
in  on  time 
performance 


Goal  =  100%  on  time 
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Audit  Results  -  Stoplight  Chart 


Raytheon 

Network  Centric  Systems 


Medium  Risk 


Low  Risk 

N/A 
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1 
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L-L 

c 

s 

Qj 

LL 

% 
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Hardware  Detailed  Design 

1 

1 

Jan-  2010 

Integration,  Verification  and  Validation 

^  1 

1 

Jan-2010 

Supplier  Selection  (General) 

17 

7 

6 

1 

1 

2 

Jan- 2010 

Measurement,  Analysis,  Improvement 

S 

5 

4 

Feb- 2010 

Hardware  Implementation 

"  4 

2 

1 

1 

Feb-2010 

Software  Implementation 

4 

1 

2 

1 

Feb- 2010 

Control  of  Design  and  Dev  Changes 

16 

8 

8 

Mar-2010 

Design  and  Development  Review 

F14 

2 

2 

1 

1 

8 

Mar-  20 10 

Engineering  Labs 

ra 

4 

21 

4 

5 

5 

6 

Mar-2010 

Program  Planning  Engineering 

14 

1 

7 

6 

Apr-2010 

Micro  Program  Review 

0 

Apr-2010 

Management  Review 

6 

2 

2 

1 

1 

Apr-2010 

Program  Planning  General 

7 

0 

3 

0 

3 

1 

May- 20 10 

Resource  Management 

3 

0 

0 

2 

0 

1 

0 

May-  20 10 

Product  and  Process  Assessment 

14 

2 

3 

3 

0 

2 

2 

2 

May- 20 10 

Production  Planning 

4 

2  | 

□ 

1  2 

0 

0 

Jun-2010 

Software  Preliminary  Design 

0 

0 

0 

0 

Jun-2010 

Engineering  Labs 

25 

4 

1 

4 

0 

8 

8 

Jun-2010 

Production  Integration  and  Test 

10 

0 

2 

0 

8 

0 

Ju  1-20 10 

DD5 

7 

4 

0 

1 

2 

Jul-2010 

Supplier  Selection  Program 

2 

0 

0 

1 

0 

1 

Ju  1-2010 

Supplier  Selection  General 

f  1 

1 

0 

0 

0 

0 

Aug- 10 

Human  Resources 

3 

0 

0 

2 

0 

1 

Aug- 10 
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Anatomy  of  the  Corrective  Action 
Timeline  Model 


Raytheon 

Network  Centric  Systems 


Open  Approval 

CA  Plan 

Open/  Containment  Complete 

\l  \  . 

CA  Close 

Approval 

CA  Verified 

CA  Complete  N. 

i  X 

id 

v 

Q A 

J  r 

^7 

q 

\J  1 

1  U 

OD 

•  There  are  four  distinct  stages  of  CA  Aging: 


Stage 

Desciiption 

Stagel A 

open  to  containment 

Stagel B 

open  to  CA  approval 

Stage2 

CA  approval  to  CA  completed 

Stage3 

CA  completed  to  CA  verified 

•  The  correlation  of  each  stage  to  the  overall  aging  of  the  record: 


Relationship 

Correlations 

Open  to  Corrective 

A-  SI  A 

0.0730 

Action  Approval 

A  -  SIB 

0.6193  ' 

explains  36%  of  the 

A  -  S2 

0.1318 

1  variation  in  cycle  time 

A  -  S3 

0.3054 

Next  Steps 


Raytheon 

Network  Centric  Systems 


•  Develop  a  process  performance  model  that  explains 
variation  in: 

-  Response  time 

-  Closure  time 

-  Containment 

-  Rejection 

•  Integrate  the  Supplier  Quality  and  Mission  Assurance 
organizations 
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NEW  Challenge  -  Integration  of  Supplier 
Quality  into  Mission  Assurance 


Raytheon 

Network  Centric  Systems 


Raytheon  Enterprise  Supplier  Assessment  (RESA)  Activity 


Network  Centric  Systems  (NCS)  RESA  Summary 


1 


250 


200 


O 

</)  150 


E  100 


50 


< 


202 


83 

1 

1 

n 

7  Wm 

□ 

2005 


2006 


2007 


HUGE  increase  in 
standardized 
supplier 

assessment  activity 


NCS  RESA  2010 

Updated  after  25th  of  the  month 


RESAs  are  tracked  monthly  with  an  annual  goal  for  suppliers  with  emphasis  on  critical 

supplier  and  leading  indicator  programs 
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NEW  Challenge  -  Integration  of 
Supplier  Quality  into  Mission  Assurance 


Raytheon 

Network  Centric  Systems 


SUPPLIER  CORRECTIVE  ACTION  REQUEST  (SCAR)  ACTIVITY 


Site 

SCARS 

Open 
On  Sept 

17 

SCARS 

Open  > 
60  Days 
On  Sept 

17 

Aug  % 
Open  > 
60  Days 

Previous 

Month  to 

Current 

Month 

Trend 

Avg 

Cycle 

Ti  me  to 

Close 

For 

SCARS 

Closed 

In  Sept 

Dallas  Lemmon 

2 

O 

0% 

27 

Ft.  Wayne 

9 

O 

0% 

46 

Fullerton  (RFO) 

2 

O 

o% 

G 

66 

Fullerton  (TRS) 

1 

O 

0% 

Goleta 

9 

3 

33% 

135 

La  rgo/StPete 

18 

14 

78% 

93 

Marl  borough 

1 

O 

0% 

37 

Mcki  nney 

19 

7 

37% 

g 

70 

Richardson  (ELCAN) 

7 

O 

0% 

Expressway 

1 

1 

100% 

91 

RSL 

4 

4 

ioo% 

Spring  Creek 

O 

O 

0% 

Waterloo 

0 

O 

0% 

20 

Total 

73 

29 

40% 

g 

65 

•Challenge:  Standardize  and  Increase  the  supplier  surveillance  presence 
and  reduce  the  supplier  corrective  action  response/closure  cycle 

•Approach: 

•  2010  Started  data  collection  with  a  goal  to  reduce  cycle  time  with  our 
suppliers 

•Moving  to  RCAS  as  a  common  tool  across  NCS  and  the  Enterprise 
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Lessons  Learned/Results 


Raytheon 

Network  Centric  Systems 


•  Dramatic  increase  in  on-time  process  evaluations 

•  Dramatic  reduction  in  CAR  closure  cycle  time 

•  The  Hawthorne  effect  really  works! 


•  Next  Step:  Develop  a  predictive  model  to  allow  us  to 
calibrate  variables  that  cause  variation  in  cycle  time 
process  noncompliance 
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Presenter  Biography-Jeff  Ricketts  systems 


Dr.  Ricketts  has  over  25  years  of  experience  in  software 
intensive  system  development  in  the  areas  of  communications, 
air  defense  and  air  traffic  control.  He  has  been  involved  in 
the  CMM/CMMI  since  it’s  inception  and  has  participated  in  12 
formal  appraisals  (SCE/SPA/SCAMPI).  He  recently  was  part  of 
the  Raytheon  NCS  hardware,  software  and  systems  engineering 
SCAMPI  effort  that  resulted  in  a  repeat  level  5  rating  for 
Raytheon’s  Network  Centric  Systems  five  major  design  centers. 
He  holds  a  Doctorate  degree  in  social  statistics  from  Washington 
State  University.  He  currently  Serves  as  the  engineering  liaison 
for  the  NCS  Program  Leadership  and  Mission  Assurance 
Organization.  22 


Presenter  Biography-Pam  Griffin 


Raytheon 

Network  Centric  Systems 


Pam  has  over  25+  years  experience  in  the  industry  in 
software  and  quality,  working  on  air  defense  training 
systems,  systems  of  systems  integration  and 
communications  systems.  She  has  been  involved  in 
CMM/CMMI  for  the  last  10  years  and  currently  serves  on 
the  NCS  Measurement  Council  and  the  Corporate 
Measurement  Community  of  Practice. 
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Navigating  the  Waters  of 
SCAMPI  Reappraisal 


NDIA  CMMI  Technology  Conference 

November  2010 


Our  Objective:  Reappraisal  Island 


Plan  Ahead 


Watch  Out  for  Hazards 


Hey,  You’ve  Done  it  Before,  Right? 


Consistency=Stability=Success 


Use  a  Quantitative  Approach 


Keep  an  Eye  on  Things 


Qihjhkri  Data 


Data  Collection 


Weekly  Points 


Esterlme'* 


AVISTA  ::  People.  Process,  Promise 
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Data 


Activity  (Al I)  | V  J 


Collection 


Earned  Value  -  Accumulated  Hours 
(AVISTA  Quality  Program:  Job  03  -  Scampi  A  2010) 


EV 

E 

-  Act  Hr 

-  Act  EV 

— A— 

-  Plan  EV 

♦ 

Milestone 

AVISTA  ::  People.  Process,  Promise. 
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Measurement  =  Key 


Reappraisal  Success! 


Data  Results 

Total  Cost  -  1 8%  below  projected  budget 

Schedule  -  Started  one  week  earlier  than  anticipated 
Mapping  Cost  -  Increased  by  1% 

Added  one  additional  appraisal  team  member 
Time  per  Appraisal  member  decreased  by  18% 

Reappraised  as  CMMI  Level  5 


r  - 


Don’t  Delay 


Questions? 


Contact  Info: 

Craig  Hale 

Process  Improvement  Manager 
Esterline  Control  Systems  -  AVISTA 
Phone  (608)  348-8815 
Email  Craiq.Hale@esterline.com 
www.  este  r  I  i  n  e .  co  m/co  n  tro  I  sy  ste  m  s/a  v  i  sta 


Esterline’* 


AVISTA  ::  People.  Process,  Promise. 
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Ester  Line- 

( Control  Systems 

( NDIA  CMMI  Conference  2010: 

Navigating  the  Waters  of  Reappraisal 

■  Gather  lessons  learned  from  key  SCAMPI  players  in  your  organization 

■  Gather  pre-SCAMPI  data  from  your  organization.  No  data?  Start  collecting! 

o  Improvement  initiatives 
o  Mapping  Activities 
o  SCAMPI  C,  B,  A 

■  Consistent  appraisal  team  lead,  and  utilize  them  for  ‘Intro  to  CMMI’  and  other  SEI  courses 

■  Consistent  key  team  members  for  appraisal  team 

■  Utilize  mini  teams 

■  Utilize  a  tool  to  capture  evidence  and  notes  (could  be  as  simple  as  a  spreadsheet  or 
database) 

■  Develop  a  model  based  on  historical  SCAMPI  related  activities  to  estimate 

■  Develop  a  company-specific  CMMI  interpretation  guide 

■  Execute  pre-SCAMPI  activities  as  a  project 

o  Utilize  DAR  to  determine  what  projects  to  use 
o  Utilize  risk  management  to  monitor  the  projects  scope  and  schedule 
o  Have  a  kickoff  with  projects  involved,  and  have  projects  provide  a  high-level  overview 
of  project 

o  Develop  an  earned  value  (EV)  plan 

o  Perform  quantitative  management  of  pre-SCAMPI  activities 
o  Monitor  and  control 

■  Performance  to  EV  plan 

■  XmR  chart  performance,  when  appropriate 
o  Have  stand  up  meetings 

o  Hold  program  reviews  with  team  and  sponsors 
o  Collect  improvement  suggestions  through  out 
o  Hold  a  project  close  out  to  identify  what  worked  well 

■  Develop  an  action  item  tracking  spreadsheet  to  track  any  evidence  questions  while 
mapping 

■  Hold  working  meeting  with  projects  in  appraisal  to  answer  evidence  questions  and  share 
information  found  in  mapping  sessions 

■  Plan  for  a  gap  analysis  a  few  months  prior  to  scheduled  SCAMPI  A 

■  Continually  improve! 
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Craig  Hale,  Process  Improvement  Manager 
Craig.Hale@esterline.com 
608-348-1048 
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Ester  line* 

( Control  Systems 

( NDIA  CMMI  Conference  2010: 

Navigating  the  Waters  of  Reappraisal 

■  Gather  lessons  learned  from  key  SCAMPI  players  in  your  organization 

■  Gather  pre-SCAMPI  data  from  your  organization.  No  data?  Start  collecting! 

o  Improvement  initiatives 
o  Mapping  Activities 
o  SCAMPI  C,  B,  A 

■  Consistent  appraisal  team  lead,  and  utilize  them  for  ‘Intro  to  CMMI’  and  other  SEI  courses 

■  Consistent  key  team  members  for  appraisal  team 

■  Utilize  mini  teams 

■  Utilize  a  tool  to  capture  evidence  and  notes  (could  be  as  simple  as  a  spreadsheet  or 
database) 

■  Develop  a  model  based  on  historical  SCAMPI  related  activities  to  estimate 

■  Develop  a  company-specific  CMMI  interpretation  guide 

■  Execute  pre-SCAMPI  activities  as  a  project 

o  Utilize  DAR  to  determine  what  projects  to  use 
o  Utilize  risk  management  to  monitor  the  projects  scope  and  schedule 
o  Have  a  kickoff  with  projects  involved,  and  have  projects  provide  a  high-level  overview 
of  project 

o  Develop  an  earned  value  (EV)  plan 

o  Perform  quantitative  management  of  pre-SCAMPI  activities 
o  Monitor  and  control 

■  Performance  to  EV  plan 

■  XmR  chart  performance,  when  appropriate 
o  Have  stand  up  meetings 

o  Hold  program  reviews  with  team  and  sponsors 
o  Collect  improvement  suggestions  through  out 
o  Hold  a  project  close  out  to  identify  what  worked  well 

■  Develop  an  action  item  tracking  spreadsheet  to  track  any  evidence  questions  while 
mapping 

■  Hold  working  meeting  with  projects  in  appraisal  to  answer  evidence  questions  and  share 
information  found  in  mapping  sessions 

■  Plan  for  a  gap  analysis  a  few  months  prior  to  scheduled  SCAMPI  A 

■  Continually  improve! 


Esterline  Control  Systems  -  AVISTA 
Craig  Hale,  Process  Improvement  Manager 
Craig.Hale@esterline.com 
608-348-1048 


©  2010,  Esterline  Technologies  Corporation.  AVISTA  is  a  registered  trademark  of  Esterline  Technologies  Corporation. 
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Tom  Fosness 

Mechanical  Systems  Engineering  Process  Improvement  Lead 
Northrop  Grumman  Corp.,  Electronic  Systems  Sector 

Rolling  Meadows 
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First  a  Brief  Story... 


Error  Prediction  and  Removal 
Process-Performance  Mode! 


BQ^es%de 


Source 


High  Maturity??? 


Drawing  Normalization 


•  Kind  of  drawing 

•  Number  of  Sheets 

•  Lines  of  Notes 

•  Drawing  Originator 

•  Drawing  Complexity 
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The  Goals  of  this  Presentation 


NORTHROP  GRUMMAN 


•  Goals 

-  Give  hardware  engineers  an  idea  of  where  to  start  when  devising  PPMs 

-  Provide  some  guidance  on  developing  PPMs 

-  Teach  tactics  and  techniques  that  hardware  engineers  can  put  directly  to  use 


•  Agenda 

-  General  guidance  on  PPMs 

-  Look  at  common  hardware  development  processes  and  identify  some  aspects  that 
lend  themselves  to  modeling 

-  Identify  what  we  want  to  predict  and  what  measures  and  baselines  are  needed  to 
establish  a  model 

-  Look  at  some  modeling  techniques  and  some  examples  of  hardware  PPMs  used  at 
Northrop  Grumman  Electronic  Systems,  Rolling  Meadows 

-  Talk  about  the  value  and  potential  benefits  of  these  models 
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Process-Performance  Models  (PPM) 


NORTHROP  GRUMMAN 


Process-Performance  Models  (PPM) 


NORTHROP  GRUMMAN 


“Models  are  defined 
to  provide  []  the  ability 
to  predict  critical 
process  and  product 
characteristics.”  [1] 
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Process-Performance  Models  (PPM) 


NORTHROP  GRUMMAN 


Controllable 

Inputs 


Must  be  statistical  or 
probabilistic  in  nature.  [4] 


“Process  and 
Product 

Measurements” 

Process 

Selections 


Predict  future  outcomes  and 
support  "what-if"  analyses.  [4] 


“Models  are  defined 
to  provide  []  the  ability 
to  predict  critical 
process  and  product 
characteristics.”  [1] 
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Process-Performance  Models  (PPM) 


NORTHROP  GRUMMAN 


Controllable 

Inputs 


Must  be  statistical  or 
probabilistic  in  nature.  [4] 


“Process  and 
Product 

Measurements” 

Process 

Selections 


Predict  future  outcomes  and 
support  "what-if"  analyses.  [4] 


“Models  are  defined 
to  provide  []  the  ability 
to  predict  critical 
process  and  product 
characteristics.”  [1] 
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Process-Performance  Models  (PPM) 


NORTHROP  GRUMMAN 


Controllable 

Inputs 


Must  be  statistical  or 
probabilistic  in  nature.  [4] 


“Process  and 
Product 

Measurements” 

Process 

Selections 


Predict  future  outcomes  and 
support  "what-if"  analyses.  [4] 


“Models  are  defined 
to  provide  []  the  ability 
to  predict  critical 
process  and  product 
characteristics.”  [1] 


PPM  Uses: 

•  Predict  Process  or  Product  Characteristics 

•  Evaluate  the  impact  of  process  composition 

•  Evaluate  the  impact  of  processes  measurements  on 

performance  relative  to  objectives 
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Process  Performance  Model  Development 


NORTHROP  GRUMMAN 


Step  2  - 

Step  1- Identify  Processes  |dentify  step  3  -  Step  4 - 

and  outcomes  of  interest  |nput  Co||ect  Ana|yze 
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What  do  Hardware  Engineers  Do? 


Requirements  Development 

•  Customer  Requirements 

•  Product  Requirements 

•  Analyze  and  Validate 


REOM 


^  Requirements  Review 
Technical  Solution 

•  Select  Component  Solutions 

•  Develop  the  Design 

•  Implement  the  Design 


^  Critical  Design  Review 
Product  Integration 

•  Prepare  for  I  ntegration 

•  Ensure  Interface  Compatibility 

•  Assemble  and  Deliver 

^  Test  Readiness  Review 


Verification 

•  Prepare  for  Verification 

•  Perform  Peer  Reviews 

•  Verify  Selected  Work  Products 


VAL 


[1] 
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What  do  Hardware  Engineers  Do?  (Translation) 


NORTHROP  GRUMMAN 


Requirements  Development 

•  Customer  Requirements 

•  Derived  Requirements 

•  Conceptual  Design 


REOM 


^  Requirements  Review 
Technical  Solution 

•  Evaluate  Design  Alternatives 

•  Model,  Analyze,  Document 

•  Procure  Hardware 


^  Critical  Design  Review 


Product  Integration 

•  Assy  Dwgs  And  Instructions 

•  Assembly  Tolerance  Analysis 

•  First-Build  Support 

^  Test  Readiness  Review 


Verification 


•  Test  Plan  and  Procedure 

•  Internal  Design  Reviews 

•  Verification  /  Qual.  Testing 


VAL 


Repetitive  Tasks 


Modeling 
Analysis 

Documentation  /  Drafting 
Work  Product  Inspections 
Work  Product  Conf.  Mgmt. 
Drawing  Revisions 
Design  Reviews 
Action  Item  Resolution 
Procurement 


Testing 
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What  do  Hardware  Engineers  Care  About? 


NORTHROP  GRUMMAN 


Hardware  Requirements 


Guiding  Design  Principles 


Size  &  Weight  (Form) 

Interfaces  (Fit) 

Performance  (Function) 

Endurance  (Environment  & 
Mission  Survivability) 

Reliability 


Design  for  Manufacture 

Design  for  Assembly 

Design  for  (Lifecycle)  Cost 

Design  for  Quality 

Process-Driven  Design 
(Consistent  &  Efficient) 

Customer  Focused  Design 


Quality  Objectives 

Meet  or  Exceed  Requirements 

Minimize  Size  and  Weight 

Maximize  Performance 

Maximize  Endurance 

Maximize  Reliability 

Maximize  Customer  Satisfaction 


Process-Performance  Objectives 
Minimize  Manufacturing  Cost 
Minimize  Assembly  Cost 
Minimize  Lifecycle  Cost 
Minimize  Escaped  Defects 
Minimize  Process  Cycle  Times 
Maximize  Customer  Satisfaction 
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Identifying  Processes  and  Measures  that  Impact 
Key  Characteristics 


NORTHROP  GRUMMAN 


Quality  Characteristics 


Process-Performance  Characteristics 


1 


1 
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Example  Questions 


NORTHROP  GRUMMAN 


•  Requirements  Development 

-  How  will  the  vibration  requirement  affect  the  product  development  time  and  cost? 

-  How  does  the  quantity  of  requirements  affect  the  development  time  and  cost? 

-  How  does  requirements  maturity  affect  task  on-time  completions? 

•  Technical  Solutions 

-  How  does  analysis  conservatism  impact  cost,  schedule,  quality?  Test  success? 

-  How  does  design  time  impact  integration,  test,  manufacture,  and  assembly  time? 

-  How  do  process  selections  to  create  prototypes  and  perform  preliminary  testing  affect 
development  cost? 

•  Product  Integration 

-  How  does  standard  part  usage,  design  reuse,  and  other  part  selections  affect 
procurement  time? 

-  How  does  the  tolerance  stack  method  affect  assembly  quality? 

-  How  does  component  testing  affect  assembly  time? 

•  Verification 

-  How  does  the  level  of  testing  affect  product  reliability? 

-  How  does  verification  method  (similarity,  analysis,  test)  affect  development  cost? 
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Example  Questions 


NORTHROP  GRUMMAN 


•  Product 

-  How  does  product  complexity  affect  schedule,  cost,  and  quality? 

-  How  do  Technical  Performance  Measures  predict  success/risk  for  the  program? 

-  How  does  design  decisions  influence  the  manufacturing  process? 

-  How  does  part  count  affect  assembly  time? 
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Decomposing  the  Process  and  Identifying 
Input  Parameters 


NORTHROP  GRUMMAN 
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Further 

Decomposition 


•  Decompose  until  sub-process  is  measureable 

•  For  each  sub-process  try  to  identify  each  of  the  following  measures: 

-  Effort  Measure  (labor  hours) 

-  Cycle  Time  Measure  (schedule) 

-  Quality  Measure  (errors  /  defects) 

-  Performance  Measure  (yield  /  efficiency) 


17 


Analysis  Time  Process  Example 


NORTHROP  GRUMMAN 


•  Analysis  Time 

•  Report  Measures 


•  For  each  subprocess  identify  measures  that  may  contribute  to  the  characteristic  of  interest 

•  Model  may  be  dependent  upon  how  one  process  or  subprocess  affects  another 
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How  will  one  process  affect  another? 


NORTHROP  GRUMMAN 


\  Effect 
Cause  \ 

Requirements 

Development 


Technical 

Solutions 


Product 
I  ntegrations 


Verification 


Requirements 

Development 

Technical 

Solutions 

Product 

1  ntegration 

Verification 

•  How  does  the 
quantity  of 
requirements  affect 
the  requirements 
development  time? 

•  How  will  reqmts. 
affect  analysis  time? 

•  How  does 
requirements 
maturity  impact  HW 
design  time? 

•  How  will 

requirements  affect 
assembly  time? 

•  How  does  the 
quantity  of 
requirements  affect 
test  cost  and 
schedule? 

•  How  does  the 
drawing  generation 
time  and  quantity  of 
errors  found  affect 
procurement  time? 

•  How  does  the 
drawing  generation 
time  and  quantity  of 
errors  found  affect 
integration  time? 

•  How  does  analysis 
time  affect  testing 
success? 

•  How  does 
prototyping  affect 
testing  success? 

•  How  does  standard 
part  usage,  design 
reuse,  and  other  part 
selections  affect 
assembly  time? 

•  How  does 
integration  time 
impact  system  testing 
time  and  number  of 
failures? 

•  How  does 
component  testing 
affect  integration 
time? 

•  How  do  design 
review  scores  impact 
testing  time? 
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Drawing  Release  Schedule  (DRS)  PPM 


NORTHROP  GRUMMAN 


•  Engineering  drawings  are  a  major  work 
product  for  hardware  engineering 

•  Once  the  design  has  been  reviewed 
and  deemed  ready  for  procurement 
drawings  are  generated,  inspected,  and 
released  to  CM,  in  series 

•  Hardware  procurement  is  dependent 
upon  the  timely  release  of  drawings 

•  Engineering,  Project  Management,  and 
Operations  are  interested  to  know 
when  drawings  will  be  released  and  to 
manage  the  Drawing  Release  Schedule 


The  DRS  PPM  forecasts  when  drawings  will  be  completed 
and  indicates  when  the  schedule  is  offtrack 
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Statistical  Tolerance  Stack  Analysis 


NORTHROP  GRUMMAN 


•  Lets  begin  with  something  familiar  to  mechanical  engineers... 


Tx=4Ta+Tb2+Tc 2 


•  Statistical  Tolerance  Stacking  allows  the  designer  to  evaluate  the 
effect  part  tolerances  will  have  on  critical  interfaces  of  a  higher  level 
assembly.  (Product  attributes  are  used  to  predict  product  attributes.) 

•  How  can  a  similar  approach  be  used  to  predict  process  performance? 
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Stack-Up  PPMs 


NORTHROP  GRUMMAN 


•  Method  works  well  for  time  durations  or  labor  hours  summing  in  series 

I_ Drawing  Release  Schedule  Total  Time  ±  TDRS 


^ - 

D1Time±Tm 

D2  Time  ±  TD2 

- w 

D3  Time  ±  TD3 

Drawing  1 

Drawing  2 

Drawing  3 

Drawing  1  +  Drawing  2  +  Drawing  3  =  Drawing  Release  Schedule 

Process  A  +  Process  B  +  Process  C  =  Composed  Process 

Iteration  1  +  Iteration  2  +  Iteration  3  =  Analysis  Time 

Test  A  +  Test  B  +  Test  C  =  Total  Test  Time 

CCA  Design  +  PWB  Layout  +  CCA  Procurement  Time 
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What  if  the  data  is  not  Normal?  (and  it  probably  is  not) 


NORTHROP  GRUMMAN 


Fractiles 


Triangular 

Distribution 


Beta 

Distribution 


Histogram 
w/  Best  Fit 


•  For  a  sufficiently  large  number  of  independent  random  variables 
that  are  identically  distributed  but  not  normal,  the  Central  Limit 
Theorem  states  that  the  resulting  summation  is  approximately 
normally  distributed,  allowing  the  RSS  approach  to  remain  valid. 
[6] 


•  Triangular  or  Beta  distributions  are  commonly  used  for  predicting 
time-based  events.  It  is  more  likely  that  an  event  will  exceed  a 
predicted  finish  date  than  finish  early  (PERT  -  Program 
Evaluation  Review  Technique).  [3] 


•  In  the  absence  of  historical  data,  a  subject  expert  may  predict 
the  best-case,  worst-case,  and  most  likely  case.  [4] 


•  Better  yet,  Process  Performance  Baselines  (PPBs) 

characterized  using  the  mean  and  standard  deviation  and  vetted 
for  special  causes  can  be  fitted  with  a  beta  distribution,  if 
appropriate. 


•  Alternatively,  PPBs  characterized  using  the  median  and  fractiles 
can  be  fitted  with  a  beta  distribution  based  upon  five  or  seven 
point  estimates.  This  approach  can  be  advantageous  because 
fractiles  are  not  significantly  impacted  by  outliers.  [5] 


•  Best  of  all,  use  Monte  Carlo  Simulation  to  predict  a  more  exact 
output  distribution  from  PPBs  with  a  variety  of  input  distributions. 
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Drawing  Release  Schedule  (DRS)  PPM 


NORTHROP  GRUMMAN 
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Drawing  Release  Schedule  (DRS)  PPM 


NORTHROP  GRUMMAN 


Printed  Wiring  Board  (PWB)  Layout  Labor 
Forecasting  PPM 


NORTHROP  GRUMMAN 


•  The  Design  Layout  Technology  (DLT) 
group  lays  out  where  components  will  be 
placed  on  a  Printed  Wiring  Board  (PWB) 
and  the  layers,  vias,  and  traces  that  will 
connect  them 

•  Historically,  the  workload  in  this  area  has 
been  difficult  to  manage  (very  up  and 
down) 

•  DLT  was  interested  in  being  able  to 
predict  their  upcoming  workload 
accurately  based  upon  board 
complexities  in  order  to  adjust 
controllable  factors  and  meet  these 
demands. 


The  PWB  Layout  PPM  forecasts  the  workload  for  the  DLT 

group  and  allows  for  the  management  of  labor 

_ 


"\ 


j 
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Generating  a  Linear  Regression  Model 


NORTHROP  GRUMMAN 


Complexity  Factors 

-  Component  Count 

-  Number  of  Nets 

-  Number  of  Pins 

-  Number  of  Vias 

-  Number  of  Connections 

-  Number  of  Vias  per  Connection 

-  Number  of  PWB  Layers 

-  Number  of  Padstacks 

Strong  correlation  discovered 
between  Labor  Hours  and  PWB 
Net  count 

Two-stage  linear  regression  model 
established 


i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i  i 

PWB  Nets 


Low  Net-Count  High  Net-Count 

o  o 
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Composite 


PWB  Layout  Labor  Forecasting  PPM 


NORTHROP  GRUMMAN 


PWB  Design  Labor 
Prediction/Estimator  Tool 


95%  Prediction  Interval 


Pro 

posed  Handling  of  PWB  Design 

Manpower  Scheduling 

No  Special  Handling 

Start  Design 

Authorize 

Overtime 

Temporarily  Hire 

Contractor 

Reschedule  Start 

Of  PWB  Design 

Comments 

Sep-09 

Oct-09 

Nov-09 

Dec-09 

Jan-10 

Feb-10 

Yes 

No 

No  / 

'  No 

DLT  and  PMO  concur  on  labor 
resolution,  board  not  in  critical 
path,  no  special  handling 
required 

1.29 

Yes 

No/ 

No 

No 

DLT  and  PMO  concur  on  labor 
resolution,  board  not  in  critical 
path,  no  special  handling 
required 

1.60 

N(/ 

/ 

Yes 

No 

No 

DLT  and  PMOconcur  on  labor 
resolution,  board  is  in  critical 
path,  special  handling  required 

4 

3.45 

/ 

No 

Yes 

Yes 

No 

DLT  and  PMO  concur  on  labor 
resolution,  board  is  in  critical 
path,  special  handling  required 

4.00 

3.83 

Yes 

No 

No 

No 

DLT  and  PMO  concur  on  labor 
resolution,  board  not  in  critical 
path,  no  special  handling 
required 

0.75 

0.76 

No 

Yes 

No 

No 

Prior  direction  provided  to  DLT 
for  handling 

1.50 

1.18 

No 

Yes 

No 

No 

Prior  direction  provided  to  DLT 
for  handling 

1.50 

1.33 

No 

Yes 

Yes 

No 

Prior  direction  provided  to  DLT 
for  handling 

1.00 

1.27 

No 

Yes 

No 

No 

DLT  and  PMO  concur  on  labor 

resolution 

1.25 

1.25 

Total  Number  Of  Man-Month* 

r 

2.89 

r 

4 

r 

9.7 

r 

9.52 

r 

3.85 

f 

0 

CCA  Board 
Number 


Number 
of  Nets 
w/Rats 


Month 
Scheduled 
To  Start 
PWB 
Design 


Overtime 
Temporary  Hire 
Reschedule 


DLT  Labor  Prediction  PPM  provides: 

•  Ability  for  Project  and  DLT  organization  to 
control  scheduling  and  costs  for  design  of 
PWBs 

•  Ability  to  predict  DLT  labor  requirements,  as 
well  as  prediction  interval 

•  Model  allows  for  feedback  of  actual  labor 
requirements  for  monitoring  and  control 
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What  do  Hardware  Engineers  Care  to  Avoid? 


NORTHROP  GRUMMAN 


http://leadinqanswers.typepad.eom/.a/6a00d834527c1469e20105368793e2970b-500wi 
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Design  for  Manufacture  and  Assembly 


NORTHROP  GRUMMAN 


•  Often  we  think  about  manufacturing  and  assembly  as  production  processes,  but 
they  are  development  processes  as  well 

-  “Use  effective  methods  to  implement  the  product  components”  [1]  (TS  SP3.1,  Subpractice  1) 
-“Assemble  product  components”  [1]  (PI  SP3.2) 

•  Best  practice  advises  that  a  component  be  designed  with  the  manufacturing  and 
assembly  processes  in  mind 

•  It  is  the  responsibility  of  the  hardware  engineer  to  consider  manufacturing  and 
assembly  costs  throughout  the  development  process  (the  earlier  the  better!) 

•  Take  advantage  of  process  baselines  from  manufacturing  focused  process 
improvement  initiatives,  such  as  Six  Sigma,  Lean,  and  Total  Quality  Management,  to 
devise  Process  Performance  Models  that  offer  insight  into  the  development  process. 


Taylor 

Shewhart 


Demming 


Six  Sigma  CMM 
ISO 


CM  Ml 


1900 


2000 
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Considerations  for  a  DFM  PPM 


NORTHROP  GRUMMAN 


•  How  do  the  manufacturing  processes  selected  during  the  design  phase  affect  the 


manufacturing  cost? 


Joining 

•  Welding 

•  Brazing 

•  Soldering 

•  Adhesive  Bonding 

•  Press  Fit 

•  Snap  Fit 

•  Mechanical 
•Fastener 

•  Other 


Machining 

•  Turning 

•  Forming 

•  Drilling 

•  Milling 

•  Sawing  &  Filing 

•  Grinding 

•  Buffing  &  Polishing 

•  ECM,  EDM,  Laser 

•  Other 


Casting 

•  Sand 

•  Shell  Mold 

•  Plaster  Mold 

•  Ceramic  Mold 

•  Investment 

•  Permanent  Mold 

•  Die  Casting 

•  Centrifugal 

•  Other 


Bulk  Deformation 

•  Rolling 

•  Drawing 

•  Extrusion 

•  Forging 

•  Other 


Powder  Metallurgy 


Sheet  Metalworking 

•  Shearing 

•  Bending 

•  Spinning 

•  Stretch  Forming 

•  Deep  Drawing 

•  Sheet  Forming 

•  Press  Forming 

•  Other 


Polymer  Processing 

•  Compression 
Molding 

•  Transfer  Molding 

•  Injection  Molding 

•  Rotational  Molding 

•  Extrusion  Molding 

•  Blow  Molding 

•  Thermoforming 

•  Other 


•  How  do  product  and  process  parameters  affect  manufacturing  process  cost? 


r 


Machining  Cost  Drivers 

•  Volume  of  Material  Removed 

•  Material  Properties 

•  Tolerances 

•  Surface  Roughness 

•  Number  of  Operations 

•  Number  of  Setups  and/or  Machines 

•  Part  Size 

•  Number  of  Non-Standard  Features 

•  Tool  Clearance 

•  Access  to  Surface  to  be  Machined 

•  Length  of  Tool  Path 


Plastic  Injection  Molding  Cost  Drivers 

•  Parting  Line 

•  Number  of  Under-cuts 

•  Nominal  Wall  Thickness 

•  Draft 

•  Surface  Finish 

•  Tolerances 


[2]  Stoll 


A  PPM  predicting  manufacturing  process  cost  would  depend 
upon  process  selections  and  product  and  process  parameters 


"A 


J 
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Considerations  for  a  DFA  PPM 


NORTHROP  GRUMMAN 


•  How  do  product  and  process  parameters  affect  assembly  process  cost? 


in  n 

Assembly  Cost  =  £  (CH  +  C7  +  Cs  +  CA  +  Cv ).  +  £  (C50  +  CF  )  . 

i= 1  7=1 

ra  =  total  number  of  parts  or  subassemblies 
n  =  total  number  of  separate  operations 
CH  =  handling  cost 
Cj  =  insertion  cost 
Cs  =  securing  cost 
CA  =  adjustment  cost 
Cv  -  verification  cost 
Cso  =  separate  operation  cost 

[2]  Stoll 


Assembly  Cost  Drivers 

•  Number  of  Parts 

•  Number  of  Fasteners 

•  Part  Commonality 

•  Standard  Part  Usage 

•  Assembly  Sequence 

•  Part  Handling 

•  Part  Insertion 

•  Part  Securing 

•  Adjustments 

•  Verification 

•  Number  of  separate  operations 

•  Tolerances 


A  PPM  predicting  assembly  process  cost  would  depend  upon 
product  and  process  cost  driving  parameters 
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Abstract  -  35  minute  presentation 


NORTHROP  GRUMMAN 


•  Looking  for  something  a  little  more... hard-hitting?  Well,  this  presentation 
is  not  for  softies  and  it  is  guaranteed  to  hit  close  to  home  with  hardware 
engineers.  Learn  tactics  and  techniques  that  hardware  engineers  can  put 
directly  to  use. 

•  We  will  begin  with  a  look  at  common  hardware  development  processes 
and  identify  some  aspects  that  lend  themselves  to  modeling. 

•  Next,  we  will  identify  what  we  want  to  predict  and  what  measures  and 
baselines  are  needed  to  establish  a  model. 

•  We  will  look  at  some  modeling  techniques  and  some  examples  of 
hardware  PPMs  used  at  the  Northrop  Grumman  Electronic  Systems 
Rolling  Meadows  Campus,  a  CMMI  Level  5  organization. 

•  Finally  we  will  talk  about  the  value  and  potential  benefits  of  these  models. 
Join  us  for  a  rock-solid  approach  to  hardware  process  performance 
models. 
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Agenda 

•  Introduction 

•  Applying  Requirements  Management  (REQM)  and  traceability 

•  Applying  Work  Planning  (WP)  and  Work  Monitoring  Control  (WMC)  to  the 
operation  of  a  services  group 

•  The  relation  between  Service  Delivery  and  WP  /  WMC 

•  Size  (attribute  estimation) 

•  Which  suppliers  to  apply  Supplier  Agreement  Management  (SAM)  to? 

•  What  is  being  audited  for  Process  and  Product  Quality  Assurance 
(PPQA)? 

•  How  to  apply  Configuration  Management  (CM)  to  service  artifacts 

•  Overlap  between  core  and  service  PAs  at  Level  2 

•  Appraisal  issues 

•  Reaction  of  a  new  group  to  the  SVC  model 

•  Suggestions  to  make  life  easier 

•  Summary 
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\ 

Just  apply  the 
GPs  to  the  PA 


What  is  SD 
GP2.2  vs.  WP 
GP2.2? 


J 


Presenting 


Appraising 


Presenting: 

-  Easy  to  gloss  over  all  the  sticky  issues  -  “Look  how  good  the  service  PAs  are.” 

Appraising: 

-  Sort  out  issues  such  as:  core  PA  interpretation,  overlap. 


©  Copyright  2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


vl.O 


3 


Introduction  -2 


THE 

PROCESS 

GROUP 


•  Appraisal  team: 

-  Three  people  experienced  with  DEV  (5-10  years)  +  LA  (22  years  with  DEV) 

-  Used  CMMI  1.2.  (This  presentation  uses  1.3  text  for  clarity.) 

•  Larger  group: 

-  200  people  that  design  and  build  large  airport  baggage  /  parcel  systems: 

»  Motors,  steel,  conveyors,  electronics,  software,  installation,  testing. 

•  Groups  appraised: 

-  Bids/Proposals  (7  people)  and  Financial  services  (11  people). 
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Services  Model  vl.2  +  1.3  -  Staged 


5  Optimizing 

4  Quantitatively 
Managed 

3  Defined 


Continuous 
Process 
Improvement 

Quantitative 
Management 

Process 
Standardization 


2  Managed 


Basic 

Project 

Management 


Org.  Performance  Management  Innovation  &  Deployment  (OPM) 
Causal  Analysis  and  Resolution  (CAR) 

Organizational  Process  Performance  (OPP) 

Quantitative  Project  Work  Management  (QWM) 

Capacity  and  Availability  Management  (CAM)  (svc) 

Incident  Resolution  and  Prevention  (IRP)  (svc) 

Service  System  Transition  (SST)  (svc) 

Service  Continuity  (SCON)  (svc) 

Service  System  Development  (SSD)  (svc,  optional) 

Strategic  Service  Management  (STSM)  (svc) 

Organizational  Process  Focus  (OPF) 

Organizational  Process  Definition  (OPD) 

Organizational  Training  (OT) 

Integrated  P^ojoot  Work  Management  (IPM) 

Risk  Management  (RSKM) 

Decision  Analysis  and  Resolution  (DAR) 

Service  Delivery  (SD)  (svc) 

Requirements  Management  (REQM) 
rWork  Planning  (WP) 

!  Work  Monitoring  and  Control  (WMC) 

Supplier  Agreement  Management  (SAM) 

Measurement  and  Analysis  (MA) 

Process  and  Product  Quality  Assurance  (PPQA) 

Configuration  Management  (CM) 


Core  PAs 
SVC  PAs 


1  Initial 


Based  on  SEI  CMMI  Services  model 
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Applying  REQM  and  Traceability 


The  purpose  of  Requirements  Management  (REQM)  is  to  manage  requirements  of 
products  and  product  components  and  to  ensure  alignment  between  those 
requirements  and  the  work  plans  and  work  products. 


•  REQM  Takes  A  LOT  of  explaining  to  a  non-familiar  services  group: 

-  The  services  groups  were  bidding/tracking  product  requirements. 

»  Caused  total  confusion  when  they  were  left  by  themselves  to  read  CMMI. 

-  Requirements  defined  as  group  roles  &  responsibilities. 

-  Initially  we  had  to  explain  why  REQM  and  SD  are  separate. 

»  Then  we  mapped  REQM  and  SD  together  -  provide  one  front  to  the  appraisal  team 
and  organization. 

-  Too  much  GP!:  Policy,  plan,  training,  monitoring  and  auditing  of  roles  & 
responsibilities  definition  activities? 

»  REQM  might  only  take  1  day  per  year  and  1  update  every  6  months. 

»  The  GPs  need  to  be  scaled  down  DRAMATICALLY  to  be  useful  OR  mapped  to  SD 
GPs.  We  merged  REQM  and  SD  GPs. 
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Explaining  REQM  in  the  Appraisal 


•  The  appraisal  team  re-wrote  the  PA  purpose  for  the  final  findings 
presentation: 

-  The  purpose  of  Requirements  Management  (REQM)  is  to  manage 
requirements  of  products  and  product  components  and  to  ensure 
alignment  between  those  requirements  and  the  work  plans  and 
work  products. 

-  The  purpose  of  Requirements  Management  (REQM)  is  to,  a)  define 
the  services  of  the  group,  b)  trace  defined  services  to  team 
activities,  c)  verify  that  resources,  service  definition  and  actual  work 
done  are  aligned,  [appraisal  team  definition] 
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Bidirectional  Traceability  (spi.4) 

SP  1 .4  Maintain  bidirectional  traceability  among  requirements  and  work  products. 

“In  a  service  environment,  you  should  be  able  to  trace  stakeholder  requirements  to  the 
elements  of  the  delivered  service  and  supporting  service  system  that  were  developed  from 
those  requirements ”  [cmmi  1.2  &  1. 3] 

Example  1 


SOX  Requirement 

Implementation 

Test 

SOX  Annex  -  section  1 

Finance  role  1,  report  1 

SOX  test  1 

SOX  Annex  -  section  2 

Finance  role  2,  report  2 

SOX  test  2 

SOX  Annex  -  section  3 

Finance  role  3,  report  3 

SOX  test  3 

Example  2 


Bid  Role 
(defined  in  SLA) 

Authority  Level 

Tasks  for  Role 

Role  1 

Approve  up  to  $X 

Obtain  estimates,  define  bid,  check  bid 

Role  2 

Approve  up  to  $Y 

Tasks  -  role  2 

Role  3 

Approve  up  to  $Z 

Tasks  -  role  3 

©  Copyright  2010  The  Process  Group.  All  rights  reserved.  www.processgroup.com  vl.O  8 
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Applying  WP  and  WMC 

•  The  group  performs  annual  resource  planning. 

•  “The  plan”  =  Annual  resource  plan  and  service-event  plan. 

•  Risks  are  assessed  monthly: 

-  E.g.,  Do  we  have  resources  to  cover  each  bid  /  financial 
report? 

•  Schedules  consist  of  bid  and  financial  report  deadlines. 

•  Stakeholders  are  defined  on  approval  and  signature  sheets. 

•  Too  much  GP!:  policy,  plan,  training,  monitoring  and 
auditing  of  annual  and  service-event  planning? 

-  Most  planning  events  were  a  few  hours  in  length. 

-  We  merged  WP  and  WMC  GPs. 

-  Audits  =  signature  approvals  with  checklists. 
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Relation  Between  SD  and  WP  /  WMC 


•  In  SD,  planning  (GP2.2)  is  used  to  plan  the  readiness  and 
operation  of  a  services  group: 

»  SD  GP2.2  -  Establish  and  maintain  the  plan  for  performing  the  process. 

»  SD  GP2.3  -  Provide  adequate  resources  for  performing  the  process . 

»  SD  GP2.4  -  Assign  responsibility  and  authority  for  performing  the  process . 

»  SD  GP2.8  -  Monitor  and  control  the  process  against  the  plan  for  performing  the  process 
and  take  appropriate  corrective  action. 

•  So  what  was  WP  /  WMC  used  for? 

-  Operations  planning,  of  which  service  delivery  is  one  significant  aspect. 

-  Annual  resource  planning  and  monthly  resource  tracking. 

-  Special  projects  (non-trivial  “other”  work). 
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Size  (Attribute  Estimation) 


•  The  group  reads  the  practice: 

-  WP  SP  1 .3  Establish  and  maintain  estimates 
of  work  product  and  task  attributes. 

-  Add  assumes  that  it  is  #Feet  of  steel, 

#motors . etc. 

•  The  bid  group  estimates  the  cost  of  a 
project: 

-  #Feet  of  steel,  #motors,  #control  panels, 
installation  labor. 


•  Luckily: 

-  Finance  already  had: 

»  Project  categorization  (the  size/complexity  of  the  project  being  financially  tracked). 

-  Bids  already  had: 

»  Bid  volume  (#bid  requests  likely  to  arrive  per  month). 
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Which  Suppliers  to  Apply  SAM  to? 

“The  scope  of  this  process  area  addresses  the  acquisition  of  products,  services, 
and  product  and  service  components  that  can  be  delivered  to  the  service's 
customer  or  included  in  a  product  or  service  system.  This  process  area’ s 
practices  can  also  be  used  for  other  purposes  that  benefit  the  service  (e.g., 
purchasing  consumables).”  [CMMI  1.3] 


•  Both  services  teams  had  no  suppliers: 

-  An  example  would  have  been  accounting  experts, 
proposal  writers. 

•  The  product  team  had  hundreds  of  suppliers;  this 
was  appraised  under  SAM  in  the  DEV  model. 
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What  is  Being  Audited  for  PPQA? 


•  Processes: 

-  Lifecycle  and  Service  Delivery  processes. 

-  Process  Area  processes. 

•  Work  products: 

-  Service  deliverables  +  critical  internal 
documents. 


Processes  audited 
via: 

•  Extensive 
Management 
Approvals. 

•  SOX  audits. 

•  Corporate  finance 
audits. 


Bid 

process 


Bid  award 


Financial 

tracking 

process 


Request  ,  | 
forbid  ^ 


£ 

Bid 


Company  Lifecycle 


& 

Monthly  $ 
reports 


ISO  audits. 
Random  quizzes. 


Documents  audited 
via: 

•  Peer  reviews  using 
checklists. 


•  Signature  approvals 
of  document  content. 

vl.O  13 
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How  to  Apply  CM  to  Service  Artifacts 


•  Identify  documents  that  service  groups  care  about,  e.g.,: 

-  Internal:  Annual  plan,  checklists,  service  agreement,  audit  results. 

-  Deliverable:  Requests,  proposals,  estimate  sheets,  contracts. 

•  Define  directory  structure  for  all  documents,  and  plan  for  archival. 

-  Merge  Data  Management  plan  in  with  CM. 

•  GPs  -  keep  them  simple! 

-  Someone  assigned  to  set  up  the  folders  and  access. 

-  Task  defined  to  “establish  CM”  that  can  be  planned/tracked. 

-  Team  meetings  for  training,  random  audits  for  objective  evaluation. 


Bid 

process 


Financial 

tracking 


Bid  award  process 


& 


Bid 


Monthly  $ 
reports 
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Overlap  Between  Core  and  Service  PAs 

*  Lots  of  overlap  -  a  MUST  fix  before  appraising,  otherwise: 

-  You  will  be  asking  the  same  question  3-4  times. 

-  The  appraisal  team  will  wear  out. 

-  The  interviewees  will  think  you  are  nuts  and  unsure  about  the  model. 


Practice  # 

REQM  Practice  Definition 

Rewrite  (Italics  =  exact  copy  of  CMMI  practice) 

r  H 

Develop  an  understanding  with  the 
requirements  providers  on  the  meaning 
of  the  requirements. 

See  SD  spl  .2 

7  12 

Obtain  commitment  to  the  requirements 
from  participants. 

See  SD  spl  .2 

r  1.3 

Manage  changes  to  the  requirements 
as  they  evolve. 

See  SD  spl  .2 

7  Ta 

Maintain  bidirectional  traceability  among 
requirements  and  work  products. 

Trace  service  requirements  to  downstream 
group  activities  and  back  (e.g.,  map  service 
descriptions  to  team  activities  and  deliverables; 
do  we  implement  our  service  definition,  do  we 
exceed  it,  do  we  have  gaps?) 

r  1.5 

Ensure  that  plans  and  work  products 
remain  aligned  with  requirements. 

Align  service  definition,  schedule/budget,  team 
activities. 

Examples: 

*  Does  the  services  list  state  to  do  A,  but  in 
reality,  the  team  is  doing  B. 

‘Analyze  measurement,  resource  and  service 
request  data. 
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Merging  Some  REQM  Practices  with  SD 


Practice  # 

SD  Practice  Definition 

Rewrite  (with  additions  from  REQM  SPs  and 
GPs.  Italics  =  exact  copy  of  CM  Ml  practice) 

1.1 

Analyze  existing  service  agreements 
and  service  data  to  prepare  for 
expected  new  agreements. 

Analyze  existing  service  agreements  and 
service  data.  Use  data  as  a  basis  for  new 
agreements.  What  can  we  achieve  based  on  the 
past  -  lessons;  overlooked  tasks?) 

(e.g . ,  In  the  past,  what  has  been  the  sales 
volume,  #bids  that  can  be  processed, 
timeliness.) 

1.2 

Establish  and  maintain  the  service 
agreement. 

Define  the  services  of  the  group  and  review  with 
staff.  [REQM  sp  1.1], 

Obtain  commitment  to  the  services  definition 
from  the  staff.  [REQM  sp  1.2] 

Establish  and  maintain  the  service  agreements 
with  customers. 

Manage  changes  to  services  (evaluate  change). 
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Overlap  (Continued) 

REQM  (GPs) 


GP  2.1 

Establish  and  maintain  an 
organizational  policy  for  planning  and 
performing  the  process. 

See  SD  GP  2.1 

GP  2.2 

Establish  and  maintain  the  plan  for 
performing  the  process. 

See  SD  GP  2.2 

GP  2.3 

Provide  adequate  resources  for 
performing  the  process,  developing  the 
work  products,  and  providing  the 
services  of  the  process. 

See  SD  GP  2.3 

GP  2.4 

Assign  responsibility  and  authority  for 
performing  the  process,  developing  the 
work  products,  and  providing  the 
services  of  the  process. 

See  SD  GP  2.4 

Practice  # 

WP  Practice  Definition 

Rewrite  (with  additions  from  WMC  GPs.  Italics  = 
exact  copy  of  CMMI  practice) 

1.1 

Establish  and  maintain  the  service 
strategy. 

Objectives:  See  MA  spl  .1 . 

Approach:  See  SD  sp  2.1. 

Risks:  See  WP  sp  2.2. 
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Appraisal  Issues 

•  Asking  CMMI  questions  to  service  professionals  unfamiliar  with 
CMMI: 

-  Plan  on  rewriting  (some  of)  the  CORE  PA  practices,  so  that  the 
interpretation  is  consistent  and  uses  easier  words. 


Practice  # 

OM  Practice  Definition 

Rewrite  (Italics  =  exact  copy  of  CMMI  practice) 

1 . 1 

Identify  the  configuration  items, 
components,  and  related  work  products 
to  be  placed  under  configuration 
management. 

Define  a  list  of  items  (files,  contracts,  bids, 
deliverables,  documents)  for  the  group  that  will 
be  placed  under  control  (e  g.,  version,  access, 
backup). 

1.2 

Establish  and  maintain  a  configuration 
management  and  change  management 
system  for  controlling  work  products. 

Establish  and  maintain  a  system  (manual  or 
automated)  to  store,  manage  and  protect  files 
on  the  list  above. 

1.3 

Oreate  or  release  baselines  for  internal 
use  and  for  delivery  to  the  customer. 

Define  naming  conventions  for  files  and 
documents,  as  they  are  created  and  released. 

2. 1 

Track  change  requests  for  the 
configuration  items. 

Track  changes  made  to  files  under  OM  control. 

Exam  pie: 

*  Track  changes  in  a  history  file  or  comment 
field  of  a  version  tool. 

2.2 

Oontrol  changes  to  the  configuration 
items. 

Establish  read/write  access  for  files  under  OM 
control . 

•  Some  service-specific  practices  need  examples  too: 

-  e.g.,  “Maintain  the  service  system”  means? 
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Reaction  of  New  Group  to  the  SVC  Model 

Target  audience  reaction: 

-  “How  does  this  relate  to  our  function?” 

»  “What  are  work  product  attributes,”  “What  risks?,”  What  are 

configuration  items?”  “What  is  a  lifecycle?” 

»  Lead  appraiser  had  to  advise  “don’ t  read  the  model,  it  will  just  confuse 
you  more.”  [specifically  the  core  PAs] 

Appraisal  team  reaction  : 

-  Initially,  total  confusion: 

»  Reading  the  model,  and  memorizing  what  each  practice  means  was 
wearing. 

»  “Model  needs  redoing,”  -  wording  and  overlap  of  practices. 

»  The  findings  of  the  class  B  appraisal  acted  as  the  new  model  definition 

for  the  team,  along  with  the  lead  appraisers  model  rewrite. 


The  core  PAs  are  the  challenge 
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Suggestions  to  Make  Life  Easier 


*  Run  a  (very)  informal  appraisal  first,  so  that: 

-  Interviewees  have  some  idea  of  what  your  model  interpretation  is. 

-  You  can  obtain  experience  asking  interview  questions  and 
understanding  the  responses. 


•  Clarify  terms  (model  glossary  might  /  might  not  help*): 

-  “Requirement,”  “stakeholders  in  planning,”  “risk”  vs.  “issue,” 
“traceability,”  “configuration  item”  “monitor  the  monitoring  process.” 


•  Train  your  team  in  the  interpretation  of  the  model  before  you  appraise: 


-  The  SVC  Supplement  class  doesn't  cover  the  core  PAs. 


-  The  Intro  to  CMMI  SVC  class  doesn’t  cover  overlap  between  PAs  and 
the  GPs  in  detail. 


*Requirement:  (1 )  A  condition  or  capability  needed  by  an  end  user  to  solve  a  problem  or 
achieve  an  objective.  (2)  A  condition  or  capability  that  must  be  met  or  possessed  by  a  product, 
service,  product  component,  or  service  component  to  satisfy  a  supplier  agreement,  standard, 
specification,  or  other  formally  imposed  documents.  (3)  A  documented  representation  of  a 
condition  or  capability  as  in  (1)  or  (2). 
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Summary 

•  The  core  Maturity  2  Level  PAs  CAN  be  used  in  a  services 
organization.  They  do  work. 

-  An  organization  at  ML2  is  a  more  organized  and  efficient: 

»  E.g.,  #Mistakes,  Response  time,  #Hours  expended. 

•  The  core  PA  practices  are  written  for  development,  not  for 
services. 

-  Intro  &  Supplement  classes  focus  on  the  benefit  of  the  service  PAs, 
not  the  difficulty  interpreting  the  core  PAs. 

-  Plan  on  a  rewrite,  otherwise: 

»  a)  You  will  have  to  explain  practices  every  time. 

»  b)  The  target  audience  will  probably  forget  the  meaning  and  get  totally  off  track. 

•  Define  what  services  or  work  the  PAs  are  being  applied  to. 
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Questions? 
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Acronyms 
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GROUP 


•  SOX  -  Sarbanes  Oxley 

•  SLA  -  Service  Level  Agreement 

•  WP  -  Work  Planning 

•  WMP  -  Work  Monitoring  and  Control 
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I've  got  a  SCAMPI  A  planned  for  a  year  from  now, 

I'm  preparing  for  a  SCAMPI  B, 
and  here  comes  VI. 3. 

What  should  my  evidence  mapping  (PHD)  look  like? 

.■^H^Strategie^r^^Chanqinq  Environment 


Sam  Fogle 

SCAMPI  High-Maturity  Lead  Appraiser 

ACE  Guides,  LLC 
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Agenda 

•  SCAMPI  VI. 3  -  When  and  What? 

•  Data  Collection?  What's  Involved? 

•  So,  what  do  I  build? 

•  Problems  we  have  met  &  Lessons  we 
have  learned 

•  Got  Any  Tips? 
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SCAMPI  VI. 3  -  When  and 
What? 


V1.3 

•  SCAMPI  VI. 3  available  Jan  2011 

•  SCAMPI  VI. 2  accepted  until  Nov  1,  2011 

•  Trying  to  cure  "PIIDs  Disease" 

•  Establishing  evidence  types  that  don't  lead  to 
wasted  work  (e.g.  distinction  of  direct/indirect) 

•  Unambiguous  rules/guidance  for  "enough?" 

•  Efficient  methods  for  collecting  needed  data 

•  "Managed  Discovery"  -  mix  of  Ver.  &  Disc. 

•  "Phased  Data  Collection"  -  evolutionary 
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Data  Collection? 

What's  Involved? 


Appraisal  Terms  and  Concepts 

•  "To  make  reasonable  judgments  regarding  an 
organization's  implemented  processes  relative 
to  the  appraisal  reference  model,  appraisal 
teams  base  their  judgments  on  the  collection 
of  objective  evidence  for  each  specific  and 
generic  practice  applicable  to  process  area 
goals  within  the  appraisal  scope."* 

•  Objective  evidence  or  "footprints",  left  behind 
after  a  practice  has  been  implemented,  are 
Practice  Implementation  Indicators  (PIIs). 

•  A  mapping  between  model  practices  and  the 
PIIs  is  called  the  Objective  Evidence  Database. 


*From  the  draft  SCAMPISM  A,  V  1.3:  Method  Definition  Document 
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Appraisal  Models 


1.  Full 

Discovery 


m 


Full  discovery  requires  that  the  team  do  a 
search  for  evidence  for  each  practice 

SCAMPIsm  pilots  took  more  than  a  month 
Appraisal  conduct  needed  to  be  streamlined 
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Appraisal  Models 


2.  Full 

Verification 


Complete 

Mapping 


Full  verification  has  the  organization  provide  a 
mapping  that  indicates  what  evidence  is 
appropriate  for  every  practice.  The 
appraisal  team  only  needs  to  search  when 
the  data  provided  is  not  clear  or  convincing 


An  Objective  Evidence  Database  could  require 
>1000  hrs  to  develop 

Appraisal  preparation  needed  to  be  streamlined 
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Appraisal  Models 


tm^ 


Managed  discovery  has  the  organization 
provide  key  artifacts  that  address  the 
majority  of  the  practices  and  the  appraisal 
team  asks  for  additional  evidence  as 
required. 

The  appraisal  conduct  may  require  additional 
time  but  preparation  is  simplified 


3.  Managed 
Discovery 


Mapping 
of  Key 
Artifacts 
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Appraisal  Models 


Comparison  of  the  three  approaches: 


Preparation  Conduct  Risk 


Full  Discovery  Low  High  High 

Full  Verification  High  Low  Low 

Managed  Discovery  Moderate  Moderate  Moderate 


SCAMPI  VI. 2  and  VI. 3  both  allow  all  of  the 
above  or  anywhere  in  between 

It  is  up  to  the  organization  and  lead  appraiser 
to  plan  compatible  preparation  and  conduct 
phases  such  that  the  organization's  needs 
are  best  met. 
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Data  Types 


VI. 2 

•  Direct  Evidence 

needed  for  every  practice 


•  Indirect  Evidence 

not  necessarily  for  EVERY 
practice* 

•  Affirmations 

not  necessarily  for  EVERY 
practice* 


*Talk  to  your  Lead  Appraiser 
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Artifacts 

needed  for  every  practice 


Affirmations 

needed  for  every  practice 


An  appropriate  set  of  artifacts? 

•  Enough  to  convince  the  appraisal  team 
that  a  practice  has  been  fully 
implemented* 

•  Multiple  instances  for  items  that  are 
produced  frequently,  e.g.  meeting  minutes 

•  Cover  the  breadth  of  the  practice 


So,  what  do  I  build? 


That's  for  you  &  your  LA  to  decide 

Balance  your  organization's  resources  and 
constraints 

•  What  are  your  organization's  priorities? 

•  What  level  of  resources  are  available? 

(Including  availability  of  CMMI  and  SCAMPI  expertise) 

•  What  level  of  risk  is  acceptable? 

You  will  need  to  create  some  form  of  Objective 
Evidence  Database  or  Mapping  that  relates 
CMMI  practices  to  your  data. 
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Who  should  build  your  mapping? 

What  kind  of  knowledge  is  needed? 

•  Understanding  of  the  model  practices 

•  Understanding  of  the  appraisal  method 

•  Understanding  of  how  the  project's  data  is 
organized 

Do  you  have  one  person  with 
If  not,  then  you  need  a  team 

•  Process  person? 

•  Project  representative? 

•  Consultant? 
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all  3? 


Mapping  Contents _ 

•  CMMI  ®  practice  name 

•  Artifact  name 

•  Artifact  location 

•  Basic  Unit  (Project)  or  Support  Group 

•  Comments  (may  explain  relevance  of 
data) 
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Mapping  format _ 

•  Does  your  Lead  Appraiser  have  a  tool 
that  they  recommend? 

•  Will  you  use  the  data  for  internal 
purposes? 

•  How  easy  is  it  to  insert  or  retrieve  data? 

•  How  easy  is  it  to  correct/update  the 
data? 

•  Will  you  map  practices  to  artifacts  or 
artifacts  to  practices? 
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Cost  &  Timing _ 

•  Objective  Evidence  Database  development 
can  be  a  major  driver  of  total  appraisal  cost 

•  The  highest  quality  will  come  from 
developing  the  Objective  Evidence  Database 
in  several  iterations,  so  start  early 

•  If  it  is  possible  to  use  the  same  sampling  in  a 
series  of  appraisals  (B,  then  A),  then  going 
through  the  B  will  identify  many  of  the 
Objective  Evidence  Database  issues  and 
allow  corrections  to  the  data  collection 
process  prior  to  the  A. 
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Validation  _ 

•  Involve  team  members  in  quality  checks 

•  Don't  wait  for  Readiness  Review  to  check  the 
quality  of  the  Objective  Evidence  Database 

•  Evolve  mapping  over  a  series  of  appraisals 

•  Effort  devoted  to  the  Objective  Evidence 
Database  (including  quality  checks)  should  be 
proportional  to  the  importance  of  achieving 
the  ratings 

Remember:  having  an  inaccurate  mapping  does  not  just 
make  it  harder  to  find  the  correct  data,  it  may  convince 
the  appraisal  team  that  appropriate  data  does  NOT  exist 


Problems  we  have  met  & 
Lessons  we  have  learned 


Problems 


•  What  sort  of  problems  have  you 
encountered  in  trying  to  plan  for  or 
develop  your  mappings? 


•  Have  others  encountered  this  same 
problem?  And  if  so,  were  you  able  to 
solve  it? 
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Lessons  Learned 


•  What  other  lessons  have  you  learned 
related  to  Objective  Evidence  Databases? 
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Got  Any  Tips? 


Potential  Work  Saver _ 

Is  the  standard  process  detailed  enough  so 
that  it  will  be  fairly  consistent  between 
implementations  where  a  specific  type  of 
data  will  be  found? 

Provide  the  units/projects  with  a  mapping 
that  already  tells  them  where  to  find  the 
evidence. 

e.g. 

PP  SP  2.2  Identify  Project  Risks 

see  section  3.4  of  the  Risk  Management  Plan 


Use  Directories 

•  When  the  evidence  you  want  to  include  is  a 
frequently  generated  item  (e.g.  monthly 
meeting  minutes  or  peer  review  reports) 
linking  to  the  directory  where  the  items  are 
stored  will  provide  advantages 

•  The  link  will  not  go  stale  -  new  items  will 
continue  to  be  populated  into  the  directory 
so  fresh  evidence  is  available 

•  The  appraisal  team  is  free  to  sample  from 
the  set,  thus  aiding  confidence  in 
institutionalization 
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Mapping  Preparation _ 

If  done  poorly 

•  Can  consume  vast  resources  to  prepare 

•  Will  reflect  a  poor  understanding  of  what  is  needed 

•  Will  cause  appraisal  to  proceed  very  slowly 

•  Can  confuse  the  true  state  of  the  practice 

If  done  well 

•  Will  require  limited  restarts  or  rework 

•  Accurately  reflects  the  work  done  in  the  organization 

•  Provides  an  efficient  means  for  an  appraisal  team  to 
find  appropriate  evidence 

•  Identifies  appraisal  risks  by  uncovering  holes  in 
implementation 
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Questions ! 
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Contact  Info _ 

•  Sam  Fogle,  Chief  Guide,  SEI-Certified 
SCAMPI  High-Maturity  Lead  Appraiser 

•  ACE  Guides,  LLC 

•  www.ACEguides.com 

•  sam@ACEquides.com 

•  240-308-0767 


®CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SMSCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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Assured  Communications^ 
Anytime.  Anywhere. 


Leaner  SCAMPI  SM 
Preparation 


Gary  Natwick 
Harris  Corporation 
16  November  2010 


NDI A  CMMI  ®  Conference  and  User  Group 


Providing  Value  To  Our  Customers 


Aviation  electronics 


Intelligence,  surveillance,  and 
reconnaissance 


Space  and  ground  satellite  Operations  and  support  services 

communications  systems 


People  -  Innovation  -  Process 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Agenda 


•  Background 

•  Challenge 

•  Approach 

•  Results 

•  Conclusion 


Leaner  SCAMPISM  Preparation 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  3 
15-18  November  2010 


Process  Improvement  Timeline  #f4/?i?#s 


Government 
Communications 
Systems  Division 
(GCSD) 


Next  SCAMPI 


CMMI-DEV+IPPD  Level  3 
AS9100  Certification 


CMMI-SE/SW  Level  3 


SW-CMM  Level  4 


ISO  9001:2000 


ISO  9001  Compliance 
SW-CMM  Level  3 

SW-CMM  Level  2 


Division  Measurement 
Handbook  (DMH) 

Process  Compliance  Monitor  (PCM) 
Integrated  Process  Manual  (IPM) 
Division  Process  Group  (DPG) 


SEI  Partner  Network 
Lean  Program 

Project  Engineering  Metrics  (PEM) 


Engineering  Process  Group  (EPG) 


Software  Engineering  Process  Group  (SEPG) 


Process  Focus 
Process  Enhancement 
Process  Advancement 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Background 


•  Process  Improvement  is  a  learned  skill 

•  To  increase  awareness  and  capabilities  Harris  is  using  a 
Lean  Six  Sigma  approach 

-  Encourage  team  members  to  look  for  ways  to  improve 
processes  by 

•  Quantifying  the  process 

•  Recommending  a  change 

•  Measuring  the  improvement 

-  Teach  skills  to  assist  in  the  efforts 

•  Lean  Fundamentals  -  eliminate  waste 

•  Simulation  -  understand  system  performance 

•  Change  behavior  -  people  skills 

•  Six  Sigma  tools  -  mathematical  skills 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Process  Compliance  Monitor  (PCM) 


Organizational  Process  Context 


Integrated  Process  Manual  (IPM) 


Processes 

T 

Statements 


Work 


Products 


Practices 

▲ 

Process  Areas 


Capability  Maturity  Model®  Integration  (CMMI®) 


(0 


J 


(0 

c 

i. 

a> 

+■* 

x 

y 


QA 

Integrated  Process 
(IP) 

Audits 

for 

IPM 


Standard  CMMI® 


Appraisal  Method 
for 

Process 

Improvement 

(SCAMPISM) 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Challenge 


•  Problem 

-  Total  cost  of  SCAMPISM  for  division  is  significant  and 
increases  every  SCAMPISM  cycle  (3-years) 

•  Goals 

-  Reduce  SCAMPISM  preparation  effort  using  Lean  method 

•  Measurement 

-  SCAMPISM  preparation  effort 

•  Benefits 

-  More  efficient  SCAMPISM  preparation  process  with  earlier 
feedback  for  corrective  actions 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Approach 


•  Objective 

-  Reduce  effort  in  conversion  for  work  products  from  internal 
organizational  requirements  to  CMMI®  Practices 

-  Establish  a  work  product  priority  to  focus  on  the  number  of 
CMMI®  practices  affected  by  each  work  product 

-  Reduce  the  rework  in  discovering  the  correct  work  product 

•  Implementation 

-  Automate  the  conversion  process 

-  Prioritized  work  product  review 

-  Utilize  process  experts  to  data  mine  for  work  products 

-  Complete  improvements  prior  to  next  SCAMPISM 

-  Establish  more  detailed  measurements  of  SCAMPISM  activities 
for  future  improvements 

•  Validation 

-  SCAMPISM 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Lean  Six  Sigma 


•  A  set  of  principles,  concepts,  and  techniques  designed  to 
enable  key  processes  to  produce  an  optimum  system 
that  we’ll  deliver  to  our  customers: 

-  Exactly  what  they  need 

-  When  they  need  it 

-  In  the  quantity  they  need 

-  In  the  right  sequence 

-  Without  defects 

-  And  at  the  lowest  possible  cost 


Leaner  SCAMPISM  Preparation 


assuredcommunications 


NDIA  CMMI®  Conference  and  User  Group  -  9 
15-18  November  2010 


Six  Sigma  DMAIC  Process 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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SIPOC 


Define 


_ i\r _ K  . _ _ K 

Measure  Analyze  Improve  Control  > 


Supplier 

-  SCAMPI 
Projects 

-  Organizational 
(HR,  DPG) 


Input 

-  SCAMPI 
Projects 

-  Program  Work 
Products 

-  Organizational 
Work 
Products 


Export,  conversion  and 
mapping  of  work  products 
is  Non-Productive 


Process 

-  Characterize 
and  reduce  the 
frequency  to 
review  each 
work  product 

-  Work  products 
into  PCM 

-  Export  from 
PCM  into  Excel 

-  Compare  deltas 
from  last  PCM 
export 

-  CMMI® 
conversion 
mapping  macro 

-  Review  &  identify 
corrective  actions 

-  Map  corrective 
actions  back  to 
PCM 


Output 

Customer 

-  Corrective  Actions 

-  Program  Team 

-  CMMI®  Progress 

-  Management 

Report 

-  Independent 

Appraiser 

Rework  in  discovering  the 
correct  work  products  is 
Non-Productive 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Brainstorming 


K 

K 

_ K 

_ K  _ K 

Define  > 

Measure  > 

Analyze  >  Improve  >  Control 

•  Facilitated  session  with  team  resulting  in  42  items 
o  Identified  3  possible  Lean  applications 

■  Reduce  effort  in  PCM  to  CMMI®  conversion  for  projects  work  products 

■  Establish  a  work  product  priority  to  focus  on  the  items  that  typically 
have  issues  and  minimize  the  amount  of  effort  appraising 

■  Reduce  the  frequency  of  discovering  correct  work  products 

0N0  detailed  measurement  breakdown  available  from  previous  SCAMPISM 
components  or  subparts 

■  Planning 
^Preparation 

S  PCM  to  CMMI®  conversion 
S  Discovery  of  work  products 
□  Review  work  products  for  corrective  action 
S  CMMI®  to  PCM  conversion 

■  Conduct 

■  Closeout 

Limited  Historical  Data  to  Demonstrate  Measureable  Improvement 


Leaner  SCAMPISM  Preparation 
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Improvement  Basis 


K  K 

K 

_ K  _ K 

Define  >  Measure^/ 

Analyze  > 

Improve  Control 

Sampled  4  months  of  SCAMPISM  effort  for  2  individuals  involved  in 
conversion  and  applied  50%  to  represent  best  estimate  of  time  spent 
o  Averaged  115  hours/month 


Work  Product  Priority 

•  Analyzed  the  number  of  CMMI®  practices  affected  by  PCM  default  work 
products  to  prioritize 

Work  Products  vs.  CMMI  Practices 


t— irvjm^-LnuDr-'-oocno*— irsiroLouDr^ooOLOuDr^orMro^-i-ocnrocx)^— 10000 
t— It— It— It— i*— i*— i*— i*— irNirMrsirMrorororororo^-^-LOLOCn 


Sample  to  Establish  Measureable  Improvement 


Leaner  SCAMPISM  Preparation 
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Improvement  Approach 


K  K  K 

K 

K 

Define  Measure^/  Analyze^/ 

Improve 

Control 

•  PCM  to  CMMI®  conversion  for  projects  work  products 

-  Reduced  to  a  one  time  event 

-  Alternative  communication  used  for  corrective  actions 

•  Establish  a  work  product  priority 

-  Focus  on  the  items  that  typically  have  issues  and  minimize  the 
amount  of  effort  appraising 

•  Find  the  correct  work  products  the  first  time 

-  Utilized  process  experts  to  data  mining  based  upon  standard 
organizational  tools  and  standard  program  directory  structure 

-  Eliminated  “bring  me  a  rock” 


Piloted  on  Next  SCAMPISM  Event 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Implement  Process  Controls 


_ K  _ K  _ K 

K 

K 

Define^  Measure^/  Analyze^  Improve^/ 

Control 

•  Let’s  Not  Do  This  Again 

-  One  time  event  for  PCM  to  CMMI®  conversion  of  projects  work  products 

-  Establish  a  work  product  priority 

-  Utilized  process  experts  to  data  mining 


•  Setup  work  codes  to  measure  SCAMPISM  activities  for  future 
improvements: 

-  Planning 

-  Preparation 

-  Program  Support 

-  Reviews  (Readiness  &  On-Site) 

-  Closeout 


Continuous  Process  Improvement 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Results 


•  Results  through  1st  SCAMPISM  Readiness  Review 
compared  to  previous  2008  SCAMPISM  event 

-  Preparation  hours  reduced  by  59% 

-  Cost  reduced  by  51% 

-  Readiness  of  work  products  was  improved 

•  Additional  benefits 

-  Reduced  rework  in  discovering  work  products 

-  Improved  consistency  in  work  products  discovered  across 
programs 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Conclusions 


•  Lean  Six  Sigma  provides 

-  Ability  to  look  at  things  differently  and  question  habits 

-  Ability  to  look  for  ways  to  improve  how  we  do  business 

-  Tools  to  enable  facilitating  change 

•  People  skills 

•  Mathematical  skills 

•  Modeling  skills 

•  Increased  awareness  of  available  resources 

-  Data  to  show  it  pays  for  itself! 


Leaner  SCAMPISM  Preparation 


assuredcommunications 
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Contact  Information 


Harris  Corporation  http://www.harris.com/ 

P.0.  Box  37  SEI  Partner 

Melbourne,  Florida  32902-0037 


Gary  Natwick 
gnatwick@harris.com 

•  SEI-Certified  Introduction  to  CMMI®  Instructor 

•  Harris  SEI  Partner  Business  &  Technical  Point  of  Contact 


®  CMMI  is  registered  with  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University. 
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Effectively  Managing  Process  Compliance 


Systems  Engineering  in  the  Face  of 
Multiple  Models,  Standards  and  Best  Practices. 


Joseph  Morin,  Integrated  System  Diagnostics 
[jfmorin@isd-inc.com] 

Erich  Meier,  Method  Park 
[Erich.Meier@methodpark.com] 


Agenda 


•  Current  Challenges 

•  Goals 

•  Proposed  Solution 

•  Tool  Support  and  Interfaces 


Benefits 
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Current  Challenges 


•  Heavy  time  and  cost  pressure  affects  product  quality 

•  Frequent  audits  or  appraisals 

•  Up  to  25%  of  work  hours  for  audit  preparation  and 
performance 

•  Assure  compliance  with  specific  models  or  standards  for 
safety,  security,  quality,  reliability,  etc.  in  parallel 
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Goals 


•  Allow  organizations  to  focus  on  the  definition  and 
implementation  of  processes 

•  Provide  automated  support  to  facilitate  enactment  of  the 
processes 

•  Provide  support  for  continuously  monitoring  adherence  to 
the  processes 

•  Support  appraisal  preparation  and  performance  by 
automating  evidence  collection 

•  Effectively  collect,  manage  and  track  non-conformances  to 
closure  in  order  to  improve  processes  and  secure  future 
audit  success 


4  A/ 1.12 
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Actively  use  the 
development  processes 
in  projects,  i.e.  by 
creating  work  products 
projects 

Link  the  actively  used 
development  processes 
to  the  models  or 
standards  that  need  to 
be  fulfilled  (e.g.  CMMI, 
ISO) 


Continuously  perform 
PPQA  activities 

Result:  evidence  data 
will  automatically  be 
collected  in  the 
background 


Process 

Improvement 


Process 

Selection 


QA  &  Appraisal 
Preparation 


Perform  Organizational 
and  Project  Processes 


Process 

Tailoring 


|  Process 
pustomization 


Assure  Process  Quality 
and  Control  Compliance 
against  Reference  Models 


Improvement 

Findings 


Appraisal 

Performance 
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Multi-model  Support 


•  Multiple  reference 
models  can  be  linked 
into  multi-models 

•  Multimodel  links  are 
weighted 

•  Multimodels  are 
specific  to  organization 
(i.e.  dependent  on 
type  of  business) 


AS  9100,  ISO  14000,  DoD  5000.01  &  02,  ... 


t  t  t 


CMMI-DEV 

CMMI-SVC 

CMMI-ACQ 

_ T _ 

_ t _ 

t 

CMMI  Cross-Constellations 

Business  and  Engineering  Processes 
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Integrated  System  Diagnostics 


Process  Composition  and  Definition 

Establish  and  maintain  Organization  Standard  Process  model(s)  in 
Model  Wizard  or  Stages  or  tooling  with  similar  capability. 

b  Create  or  import  a  model  of  your  enterprise  processes  that  incorporates  the 
desired  best  practices  of  applicable  standards  and  model(s).  Update  when 
needed. 

□  This  step  is  performed  once  for  each  level  at  which  the  process  is  tailored 
within  the  enterprise,  (e.g.  Enterprise,  Division,  Department,  Product  Line) 

Ultimately  it  is  tailored  to  a  particular  project .  At  that  point  it  becomes  a 
representation  of  the  Project’s  Defined  Process. 

□  It  can  then  be  used  as  the  basis  for  identifying  and  analyzing  information 
and  evidence  supporting: 

□  Project  QA  activity  and  monitoring, 

°  Internal  compliance  audits,  and 
a  External  benchmarking  appraisals  or  compliance  audits. 
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QA  Audit  and  Appraisasl  Demonstration 


7 


Integrated 

System 

Diagnostics 

INCORPORATED 


Relationship 

Enterprise  Architecture,  Best  Practices,  Process  Improvement 


Activity  A- Missing  Process 


Activity  B  -  Needs  Process  Improvement 


ISF  CPP  X  Objective  Z 
ISO  Clause  B 
CMMI  Practice  SP  #.# 


Process  Improvement 
Implementation  Guidance 


Best  Practices 


®  2005,  Appraisal  Wizard,  ISF  for  Excellence  -  AH  rights  reserved  -  ISO  Inc. 


8 


# 


Integrated 


Diagnostics 

INCORPORATED 


Relationships: 
Process  Architecture, 
Best  Practices,  Audits 


ISF  for  Excellence 


Best  Practices 


Integration  /  System 


Market  & 
Competitors 


Organ  rza  (In  aJ 


Human 

Capital 


Culture,  Strategic 
Ali  eminent 

Infrastructure  & 


Innovation  and 


Customers 

(pre-contract) 


r  | 

r 

|  P&S  Execution 

P&S  Sustentataorj 

i 

^^Managemen^^ 

i- 

r 

i 

Supplie^^^ 

i 

S' 

Customers 

(post-contract) 


Process  Definition 


Customer  Process  Architecture 


Process  Compliance 

Diagnostic  Tools 


J 


Global  Knowledge  Management  Solution 


■ 


Project 

Driven 

WorK 


Eno-to-t  nci  Delivery 

ISO 


Business 

Process 

Support 


aoh  wiiir  injd  jw^eca 


1 


eSCM 

m  Lhe  fl.nl  Ire  npB-nlrum  ol  Itis  unlrul  toureJng  [llBcyel* 


PPQA  /  Audit  /  Appraisal  Findings 


■airJtta 


Enterprise  Appraisal 


2005,  Appraisal  Wizard,  ISF  for  Excellence  -  AH  rights  reserved  -  ISD  Inc. 


\SF  for (Exceflence  -  Systemic  View  Integration  Framework 


Solution  Details 


Reference  models  are 
broken  into  different 
model  requirements 

Process  elements  are 
linked  with  model 
requirements 

Model  requirements 
are  mapped  to 
evidences 

Evidence  data  maps 
can  be  generated 
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Example:  Process  Lifecycle 


■i  stages 

Software  Engineering  >  Phases 

Home  |  Settings  i.  Notification  i  Help  J  Info  k  Logout  Welcome  Superuser 

|  Issues  '  At  a  glance  Info  center  Down  lea  dcenter  Q 

Wiki  |  Projects  |  Users 

j  Search 


Go 


j  Project 


b 


Method  Park 
Standard  Processes 
Software  Engineering 


j  Process 


Overview 

Activities 

Phases 

Documents 

Roles 

Methods 

Tools 

Metrics 

Trainings 

Guided  Tour 

Resources 

Index 


j  Quickfist 


i  &  1 1 


Development  Projects 
Software  Engineering 
Software  Project 
Systems  Engineering 


Actions 

Phases 


□  ?  .S3  i  a 


j  Service 


Create  CR 
Homepage 
Mail  Webmaster 


Version  El  a  selin e_X,  -■■.--easf 


http  ://10 .10 .100 .10  :S0S0/p  kit/p  ro  c  es  s/vi  ewEI  em  ent.  d  o  ?i  d = 3054  &ty  p  e=  p  h  a  5  e 


Internet  |  Geschutzter  Modus:  Aktiv 


- 
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Example:  Process  Compliance  Mapping 


ii  stai 

ges 

Software  Project 

Home  5 

i  Settings  >  Notification 

i  Help  i  Info  |  Logout 

Welcome  Superuser 

Issues 

At  a  glance 

In  fn  center 

|  Do  wn  load  center 

Process  Workbench 

Wiki 

Projects 

Users 

j  Search 


Go  ( 


j  Project 

L*LJ 

Method  Park 

Development  Projects 

Project  A 

Software  Project 

ij  Process 

Overview 

Activities 

Phases 

Documents 

Roles 

Methods 

Tools 

Metrics 

Trainings 

Guided  Tour 

Resources 

Index 

j  Ouicklist 

■fr 

Development  Projects 

P 

Software  Engineering 

0 

Software  Project 

m 

Systems  Engineering 

m 

J  Service 

Create  CR 
Homepage 
Mail  Webmaster 


+-n  □  0 

CMMI-DEV  continuous  representation 

Name 

Rating 

S  Activities 

M  1 

E)  Project  Management 

BPP:  Project  Planning 

iT  SG  1:  Establish  Estimates 

El  SG  2:  Develop  a  Project  Plan 

SP2.1:  Establish  the  Budget  and  Schedule 

'X'm- 

SP  2.2:  Identify  Project  Risks 

k  1 

SP  2.3:  Plan  for  Data  Management 

SP2.4:  Plan  for  Project  Resources 

itkir 

SP2.5:  Plan  for  Needed  Knowledge  and  Skills 

SP2.6:  Plan  Stakeholder  Involvement 

SP2.7:  Establish  the  Project  Plan 

ffi  SG  3:  Obtain  Commitment  to  the  Plan 

#*☆ 

i±i  GG  1 :  Achieve  Specific  Goals 

EGG  2:  Institutionalize  a  Managed  Process 

EGG 3:  Institutionalize  a  Defined  Process 

l+lGG  4:  Institutionalize  a  Quantitatively  Managed  Process 

EGGS:  Institutionalize  an  Optimizing  Process 

E  Documents 

Description 


SR  2,1:  Establish  the  Budget  and  Schedule 


Establish  and  maintain  the  project's  budget  and  schedule. 

The  project’s  budget  and  schedule  are  based  on  the  developed  estimates  and  ensure  that  budget 
allocation,  task  complexity,  and  task  dependencies  are  appropriately  addressed. 

Event-driven,  resource-limited  schedules  have  proven  to  be  effective  in  dealing  with  project  risk. 
Identifying  accomplishments  to  be  demonstrated  before  initiation  of  the  event  provides  some  flexibility  in 
the  timing  of  the  event,  a  common  understanding  of  what  is  expected,  a  better  vision  of  the  state  of  the 
project,  and  a  more  accurate  status  of  the  projects  tasks. 

Subpractices 

1.  Identify  major  milestones. 


: 


Assignments 


Rating 


m  0  Activities.  t 
%  Name 
B  Activities 

E1  Requirements  Management 
E  Project  Management 
50  Establish  Project  Plan 

Review  Project  Plan 
Create  Status  Report 
El  System  &  Software  Design 
E  Quality  Management 
E  Verification  &  Validation 
E  Configuration  Management 
E  Project  Quality  Assurance 


ET  Commentary 


Percentage  30| 
View  description 
Edit  properties 
Cancel  assignment 


Fertig 


4^'  Internet  |  Geschutzter  Modus:  Alctiv 


-a  -  %100%  - 
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Example:  Compliance  Gap  Analysis 


■J  sta 

ges 

Software  Ft  eject 

Home  | 

i  Settings  i  notification 

i  Help  i  Info  1 

Logout 

Welcome  Superuser 

Issues 

At  a  glance 

Info  center 

|  Oownloadcenter 

|  Process  Workbench 

Wiki 

Projects 

Users 

“l  ™ 

j  Search 


Go  1 


j  Project 

Method  Park 

Development  Projects 

Project  A 

Software  Project 

ij  Process 

Overview 

Activities 

Phases 

Documents 

Roles 

Methods 

Tools 

Metrics 

Trainings 

Guided  Tour 

Resources 

Index 

j  CiuickJist 

* 

Development  Projects 

0 

Software  Engineering 

0 

Software  Project 

0 

Systems  Engineering 

0 

j  Service 

Create  CR 
Homepage 
Mail  Webmaster 


□  tk  H 


Perform  Gap  Analysis 


Gaps  between  CMMI-DEV  continuous  representation  and  Software  Project 

Name 

% 

Rating 

0  Activities 

0  Project  Management 

B  PP:  Project  Planning 

0% 

0  SG  1:  Establish  Estimates 

0% 

0  SP  1 .2:  Establish  Estimates  of  Work  Product  and  Task  Attributes 

50% 

0  Assignments 

Activity:  Estabiish  Project  Plan 

50% 

0  SG  2:  Develop  a  Project  Plan 

0% 

0SP  2.1 

Establish  the  Budget  and  Schedule 

50% 

kk-  ? 

SP  2.3 

Plan  for  Data  Management 

0% 

a  SP  2.4 

Plan  for  Project  Resources 

50% 

kk?- 

SP  2.5 

Plan  for  Needed  Knowledge  and  Skills 

6% 

SP  2.6 

Plan  Stakeholder  Involvement 

6% 

0  SP  2.7 

Establish  the  Project  Plan 

56% 

kk:\ 

0  SG  3:  Obtain  Commitment  to  the  Plan 

0% 

SP  3.3:  Obtain  Plan  Commitment 

0% 

i+i  GG  1 :  Achieve  Specific  Goals 

0% 

0GG  2:  Institutionalize  a  Managed  Process 

0% 

0GG3:  Institutionalize  a  Defined  Process 

0% 

0GG  A;  Institutionalize  a  Quantitatively  Managed  Process 

6% 

©GG5:  Institutionalize  an  Optimizing  Process 

6% 

0  Documents 

□  Back  to  Compliance  Workbench 


Fertig 


4-'  Internet  |  Geschutzter  Modus:  Aktiv 


i  -  \100%  - 
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Example:  Evidence  Data  Generation 


1^]  Microsoft  Excel  -  pi"id£l]  3^ 


|c£j  Datei  Bearbeiten  Ansicht  Einfugen  Format  Extras  Daten  Fenster  ? 

=  a  is  y  ji.cj  *  *  ar  /!  i  'lima  @  t 

|  i  Arial  T  10  ^  jj 

0 

K  U 

PI 

Frage  hker  eingeben  ^  _  fi1  X 

=  =  1  43  %  OQO  c  |  w  W  1  El  T  &  -  A  T  1 

i  ^Snaglt  £['  |  Window 

A1  -  f*  # 


A 

B 

C 

D 

E 

F 

G 

H 

1 

J 

K 

1 

# 

PA 

GoallD 

PracticelD 

Practice 

Project 

Evidence  Type 

Document  Title 

Link 

Doc-record  Comments 

Comments 

2 

l' 

PP 

SC  1 

SP1.1 

SP  1.1:  Estimate  the  Scope  af  the  Project 

Software  Project 

D 

Software  Project  Plan 

Process 

3 

2 

PP 

SC  1 

SP1.1 

SP  1 .1 :  Estimate  the  Scope  of  the  Project 

Software  Project 

D 

Project  Plan 

4 

3 

PP 

SC  1 

SP1.1 

SP  1.1:  Estimate  the  Scope  of  the  Project 

Software  Project 

D 

Software  Project  Plan 

5 

4 

PP 

SCI 

SP1.1 

SP  1.1:  E  stimate  th  e  Sco  pe  o  f  the  Project 

Software  Project 

D 

Establish  Project  Plan 

Process 

6 

5 

PP 

SC  1 

SP1.2 

SP  1 .2:  Establish  Estimates  of  Work  Product  and  Task  Attributes 

Software  Project 

D 

Establish  Project  Plan 

Process 

7 

6 

PP 

SC  1 

SP1.3 

SP  1 .3:  Define  Project  Lifecycle 

Software  Project 

D 

Stages  Process  Definition 

See  lifecycle  definition  of  the  project  itself  in  Stages. 

8 

7 

PP 

SC  1 

SP1.4 

SP  1.4:  Determine  Estimates  of  Effort  and  Cost 

Software  Project 

D 

Create  Quality  Plan 

Process 

9 

3 

PP 

SGI 

SP1.4 

SP  1.4:  Determine  Estimates  of  Effort  and  Cost 

Software  Project 

D 

Review  Quality  Plan 

Process 

10 

9 

PP 

SC  2 

SP2.1 

SP2.1:  Establish  the  Budget  and  Schedule 

Software  Project 

D 

Establish  Project  Plan 

Process 

11 

10 

PP 

SC  2 

SP2.2 

SP  2.2:  Identify  Project  Risks 

Software  Project 

D 

Risk  Analysis 

Process 

12 

11 

PP 

SC  2 

SP2.2 

SP2.2:  Identify  Project  Risks 

Software  Project 

D 

Risk  Analysis 

13 

12 

PP 

SC  2 

SP2.2 

SP  2.2:  Identify  Project  Risks 

Software  Project 

D 

Software  Project  Plan 

Process 

14 

13 

PP 

SC  2 

SP2.2 

SP2.2:  Identify  Project  Risks 

Software  Project 

D 

Project  Plan 

File 

15 

14 

PP 

SC  2 

SP2.2 

SP  2.2:  Identify  Project  Risks 

Software  Project 

0 

Software  Project  Plan 

16 

13 

PP 

SC  2 

SP2.2 

SP2.2:  Identify  Project  Risks 

Software  Project 

D 

Establish  Risk  Analysis 

Process 

17 

16 

PP 

SC  2 

SP2.2 

SP  2.2:  Identify  Project  Risks 

Software  Project 

D 

Establish  Project  Plan 

Process 

18 

17 

PP 

SC  2 

SP2.2 

SP2.2:  Identify  Project  Risks 

Software  Project 

D 

Review  Project  Plan 

Process 

19 

IS 

PP 

SC  2 

SP2.4 

SP2.4:  Plan  for  Project  Resources 

Software  Project 

D 

Establish  Project  Plan 

Process 

20 

19 

PP 

SC  2 

SP2.7 

SP2.7:  Establish  the  Project  Plan 

Software  Project 

D 

Risk  Analysis 

Process 

21 

20 

PP 

SC  2 

SP2.7 

SP  2.7:  Establish  the  Project  Plan 

Software  Project 

D 

Risk  Analysis 

22 

21 

PP 

SC  2 

SP2.7 

SP2.7:  Establish  the  Project  Plan 

Software  Project 

D 

Establish  Project  Plan 

Process 

23 

22 

PP 

SC  3 

SP  3.1 

SP  3.1:  Review  Plans  that  Affect  the  Project 

Software  Project 

D 

Review  Project  Plan 

Process 

24 

23 

PP 

SC  3 

SP  3.1 

SP3.1:  Review  Plans  that  Affect  the  Project 

Software  Project 

D 

Review  Protocol 

Process 

25 

24 

PP 

SC  3 

SP  3.1 

SP  3.1 :  Review  Plans  that  Affect  the  Project 

Software  Project 

D 

Review  Analysis 

26 

23 

PP 

SC  3 

SP  3.1 

SP  3.1:  Review  Plans  that  Affect  the  Project 

Software  Project 

D 

Review  Record 

27 

26 

PP 

SC  3 

SP  3.2 

SP  3.2:  Reconcile  Work  and  Resource  Levels 

Software  Project 

D 

Review  Project  Plan 

Process 

28 


29 

JO 

31 

32 

33 

34 

TiC  _  _ _ _ 

n  ^  ►  n\piidl/  \*  _  _ _ _ m  |  y  \ 
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Integrated  System  Diagnostics  Audits  and  Appraisa|  Process 


i  )it.L 

>> _ X'  -X*  -H 


Use  Model  Wizard  to  create  or  import  a 
model  that  represents  your  Organizational 
Standard  Model  and/or  the  desired  QA 
Reference  Model. 


1 


Use  the  AW  tool  during  the  audit  to  document 
the  audit  finding(s)  (e.g.  “Compliance,  “Non- 
Compliance  (N/C)”,  “Information  Needed” 
record  types). 

•Write  up  N/C  Action  Item  records 
•Determine  compliance  ratings 


Use  Appraisal  Wizard  (AW)  to  setup  Appraisal  Wizard 
Audit  Template(s)  for  each  type  of  Audit  you  want  to 
perform  (e.g.  setup  unique  record  types,  status  values, 
and  document  types,  etc.) 

•Create  Audit  Question  records  to  build 
audit  checklist  for  set  of  audit 
checkpoints. 

•Use  the  Record  Documents  tab  and 
the  document  list  to  identify  the 
expected  objective  evidence  for  each 
audit  question. 


/ 

\  .//  .4 


■c\ 


I 


•(/- 


4 


-0- 


Use  the  AW  tool  to  build  various  reports  of  the 
audit  finding(s)  (e.g.  Reports  of  %  compliant/% 
non-compliant  findings;  List  N/C  Action  Items; 
etc.)  to  prepare  for  follow-on  audits. 


JL 


•Use  Model  Mapper  to  map  QA  Reference  model  to  a  standard  or  model  (e.g.  CMMI)  for  use  in  appraisals. 
•Import  new  “Mapped  Model”  into  AW  to  use  audit  results  and  organization's  existing  data  to  support 
Readiness  Reviews  and  Appraisals. 
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QA  Audit  and  Appraisasl  Demonstration 
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Integrated  System  Diagnostics 


QA  Reference  Model  in  Model  Wizard 


Model  Elements  (MM002)  -  Copy  of  Kingland  Systems  Corp  (Demo)_081106 


File  Edit  Model  Element  View  Text 


l3  ■  ■  ■  Copy  of  Kingland  Systems  Corp  (Demo)_081106 

I . ^  Pofcy 

Organizational  Procedures 
£)•••<§  I  Requirements  Management 

1.0  Plan  for  Requirements  Management 

f . «  1. 1  Assign  Resources 

1. 2  Establish  Customer  Understanding  Expectations 

i . «  1.3  Plan  for  Requirements  Changes 

'  -  1.4  Finalize  Requirements  Management  Planning 

£}■■  2.0  Establish  an  Initial  Requirements  Baseline 

*  i  Jt  ^  •  «  • 

2. 1  Determine  Requirements  Needs 

2.2  Propose  Initial  Requirements  Baseline 

2. 3  Achieve  Initial  Customer  Understanding 

; . «  2.4  Establish  Initial  Requirements  Baseline 

Jj-  i^i  3.0  Maintain  a  Requirements  Baseline 
: . «  3. 1  Complete  Requirements  Baseline 

3. 2  Confirm  Customer  Understanding 

3.3  Establish  Requirements  Baseline 
3  ^  4.0  Manage  Requirements  Change  and  Inconsistency 

4. 1  Review  Requirements  Change  Requests 


- ^  |4. 2  Monitor  for  Requirements  Inconsistencies 


Element  Name: 


[4.2  Monitor  for  Requirements  Inconsistencies 


Element  Abbreviation: 


4.2  Monitor  for  Requirements  Inconsistencies 


Element  Type: 


j  0  rganizational  Procedure  S  ub-S  tep  ▼ 

|  Full  Description  |  Short  Description  Mapping 


Step  4.2.1:  To  prepare  for  monitoring  activities,  review  current  Internal  and  Customer  Requirements  Baselines  to  ensure  an  accurate  and  l. 
Method:  N/A 

Step  4.2.2:  Meet  with  individuals  who  maintain  the  requirements  traceability  matrix  to  identify  any  inconsistencies  between 
Business  Requirements  and  Solution  Requirements 
Requirements  and  WBS  assignment 
Requirements  and  Solution  Design 
Requirements  and  Test  Cases 
Requirements  and  Functions/Features 

The  goal  of  this  meeting  is  to  ensure  that  the  requirements  traceability  matrix  is  accurate.  Examples  of  inconcistencies  include: 

A  new  requirement  was  added  yet  no  design  was  created 
A  requirement  changed  and  a  test  case  was  not  updated 

A  function/feature  is  being  modified  as  a  part  of  the  project,  yet  is  not  traced  to  a  requirement 
Generate  a  requirements  change  request  for  any  inconsistencies  identified  during  this  meeting. 

Method:  CMMI  Direct  Evidence 

Meeting  minutes  w/  attendees  and  notes  specifying  requirementsinconsistencies  review  activities 
Requirements  change  requests 

Step  4.2.3:  Meet  with  Project  leadership  to  review  Project  Plans  to  identify  any  inconsistencies  between  Requirements  Baselines  and  Proj< 
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Model  KSC  »  Element  Type  [ALL]  ^  Rating  Level  TBD 
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:  Record  Filter  f|C 


Element  Name 

&■  -2  Kingland  Systems  Corp  (REQM  Demo)_081 1 06 
I  Policy 

3  ^  Organizational  Procedures 

^  Requirements  Management 
d  1 .0  Plan  for  Requirements  Management 

1.1  Assign  Resources 

! .2  Establish  Customer  Understanding  Expectations 

1 .3  Plan  for  Requirements  Changes 

1 .4  Finalize  Requirements  Management  Planning 
id  2.0  Establish  an  Initial  Requirements  Baseline 

'S1  2.1  Determine  Requirements  Needs 
^  2.2  Propose  Initial  Requirements  Baseline 

2.3  Achieve  Initial  Customer  Understanding 

2.4  Establish  Initial  Requirements  Baseline 
id  ^  3.0  Maintain  a  Requirements  Baseline 

^  3.1  Complete  Requirements  Baseline 
^  3.2  Confirm  Customer  Understanding 

3.3  Establish  Requirements  Baseline 
d  \\  4.0  Manage  Requirements  Change  and  Inconsistency 
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Step  1.2.1:  Document  all  parties  that  will  serve  in  the  Customer  role.  Requirements 
are  considered  valid  if  they  are  understood  from  the  Customer. 

Method:  Stakeholder  Analysis 

Step  1.2.2:  For  step  1.2.3  through  1.2.4,  communicate  with  the  Customer(s). 
Method:  N/A 
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ID  Audit  Checklist  Type 

Ree  Status 

Doc  ID  Ev  Type 

Title 

1 495 

1 .  Review  and  confirm  assignment  of  the  Requirements  Lead  role  with  the  project  representative  . 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  or  their  project  library. 

3.  Identity  the  objective  evidence  provided  as  direct  or  indirect. 

Audit  Checklist  Item 

Candidate 

Tiooi 

Expected  Direct  Evidence 

Project  Planning  Timeline 

r1 497 

1 .  Determine  if  the  Customer  responsible  for  requiremetns  interface  has  been  identified. 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identify  the  objective  evidence  provided  as  direct  or  indirect. 

Audit  Checklist  Item 

Candidate 

*0005 

Expected  Direct  Evidence 

Stakeholder  Analysis 

"1 499 

1 .  Has  the  project  documented  the  way  it  captures  requirements  change  requests  from  the  Customer  and 
other  stakeholders? 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identity  the  objective  evidence  provided  as  direct  or  indirect. 

Audit  Checklist  Item 

Candidate 

'0002 

Expected  Direct  Evidence 

Requirements  Management  Plan 

"1509 

1 .  Determine  who  on  the  team  is  responsible  for  maintaining  the  respective  portions  of  the  traceabilily 
matrix. 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identify  the  objective  evidence  provided  as  direct  or  indirect. 
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Requirements  Management  Plan 
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1 .  Confirm  assigned  resources  have  the  requisite  skills  or  are  planned  for  training  required  to  perform  the 
roles  and  responsibilities. 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identify  the  objective  evidence  provided  as  direct  or  indirect. 
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|1.  Deteminethe  communication  that  planned  orthose  that  have  occurred  for  establishing  an  understanding 
of  requirements  with  customers. 

2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identify  the  objective  evidence  provided  as  direct  or  indirect. 
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1 2.  Collect  the  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identity  the  objective  evidence  provided  as  direct  or  indirect. 

Audit  Checklist  Item 
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'0002 

Expected  Direct  Evidence 

Requirements  Management  Plan 

"1 51 3 

1 .  Has  a  specific  method  for  achieving  an  understanding  of  requirements  been  identified  by  the  project? 

2.  Collectthe  expected  direct  evidence  from  the  project  respresentative  ortheir  project  library. 

3.  Identity  the  objective  evidence  provided  as  director  indirect. 
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Element  Name 

J  Kingland  Systems  Corp  (REQM  Demo)_081 106 
i  ^  Policy 

d  Organizational  Procedures 
d  Q  Requirements  Management 

d  ^  1 .0  Plan  for  Requirements  Management 
1.1  Assign  Resources 
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Step  1.2.1:  Document  all  parties  that  will  serve  in  the  Customer  role.  Requirements  : 
are  considered  valid  if  they  are  understood  from  the  Customer. 

Method:  Stakeholder  Analysis 

Step  1.2.2:  For  step  1.2.3  through  1.2.4,  communicate  with  the  Customer(s). 
Method:  N/A 
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d  3.0  Maintain  a  Requirements  Baseline 

3.1  Complete  Requirements  Baseline 

3.2  Confirm  Customer  Understanding 
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Benefits 


Reduce  complexity 

•  Process  descriptions  become  consistent  and  lean 

•  No  „CMMI  or  ISO  speak" 

•  Fulfill  multiple  standards  in  parallel 

Reduce  efforts  and  costs 

•  Companies  reported  up  to  60%  less  efforts 
for  audit  preparations 

•  No  interruption  of  operational  work  because 
of  audit  preparations 

Concentrate  on  process  improvement, 
not  process  administration 
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CMMI®  VI  .3:  The  Rest  of  the  Story: 
Model  Changes  at  CL/ML  1-3 

November  16  2010 

1  1  V  V  V  III  MV  1  1  \J  y  fcV  1  V 

VI. 3  Core  Model  Team  (with  Mike  K., 
as  spokesperson) 

Software  Engineering  Institute 

Carnegie  Mellon  University 

®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 

are  Engineering  Institute  Carnegie  Mellon  c2010camegieM.1ionuniv.rsKy 

CMMI  Models  for  Three  Constellations 


Core  PAs  appear  in  all  CMMI 
models;  however... 

•  These  PAs  are  not  identical 
across  all  models. 

•  Informative  material  can  be 
different  so  that  users  interpret 
goals  and  practices  for  their 
area  of  interest. 


CMMI-DEV 

CMMI-DEV  provides 
guidance  for  measuring, 
monitoring  and  managing 
development  processes. 

v 


Material  that  is  the  same  is  called  CMMI  Model 
Foundation  (CMF). 

Sometimes  practices  can  be  different  in  one  model 
from  another  (i.e..,  PP  &  PMC). 

There  is  one  Shared  PA  (SAM). 


CMMI-SVC 


i 


CMMI-SVC  provides 
guidance  for  those 
providing  services  within 
organizations  and  to 
external  customers. 


CMMI-ACQ 

CMMI-ACQ  provides 
guidance  to  enable 
informed  and  decisive 
acquisition  leadership. 

M  W 
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Process  Area  Components 
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Continuous  Representation:  PAs  by  Categories 
(And  Potentially  Across  Constellations) 


Service  Establishment 

[Product] 

Acquisition 

and  Delivery 

Engineering 

Engineering 

I  REQM 


*CMMI-SVC  only 


AM,  SSAD 


Process 

Project  and  Work* 

Support 

Management 

Management 
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Summary  of  Generic  Goals  and  Practices 


Generic  Goals 

Generic  Practices 

GG1:  Achieve 

Specific  Goals 

GP  1.1 

Perform  Specific  Practices 

GG2:  Institutionalize  a 

GP2.1 

Establish  an  Organizational  Policy 

Managed  Process 

GP2.2 

Plan  the  Process 

GP2.3 

Provide  Resources 

GP2.4 

Assign  Responsibility 

GP2.5 

Train  People 

GP2.6 

Manage  Configurations 

GP2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP2.8 

Monitor  and  Control  the  Process 

GP2.9 

Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Hiaher  Level  Manaaement 

GG3:  Institutionalize  a 

GP3.1 

Establish  a  Defined  Process 

Defined  Process 

GP3.2 

Collect  Improvement  Information 

Adapted  from 
Cepeda  Systems  & 
Software  Analysis,  Inc. 


CL  4-5,  GGs  4-5,  GP  4.*  and  GP  5* 
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The  Problem 

As  VI  .2  models  were  created  and  released,  they  became  out  of  synch  with 
one  another.  Improvements  made  in  one  model  were  not  made  in  others 
simply  because  of  differences  in  release  schedules. 

Overview  of  Solution 

Analyzed  differences  among  the  three  models  (ACQ,  DEV,  SVC)  to  identify 
opportunities  to  improve  all  three  while  improving  commonality.  Examples 
of  improvements  include  the  following: 

•  GGs,  GPs,  and  GP  elaborations  consolidated  into  one  location  (DEV) 

•  Improved  measurement,  supplier,  and  agreement  terminology  (DEV) 

•  Improved  definitions  of  terms  related  to  products  and  services  (DEV,  ACQ) 

•  Increased  emphasis  on  customer  satisfaction  (all  three) 

•  Improved  examples,  example  work  products,  and  notes  (all  three) 

Many  of  the  specific  changes  made  to  the  model  are  due  to  harmonization 
and  are  covered  in  the  PA  Changes  modules  of  Upgrade  Training. 
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Teaming  Concepts., 


The  Problem 

Teams  are  clearly  relevant  to  product  development.  How  teams  are  established  in 
an  organization  has  a  lot  to  do  with  whether  or  not  they  are  successful. 
However,  there  are  no  specific  practices  addressing  rules  for  and  establishing 
teams  in  DEV,  instead  there  is  the  Integrated  Process  and  Product 
Development  (IPPD)  addition,  which  is  optional.  Fewer  than  5%  of  recent 
appraisals  have  included  IPPD. 

For  acquisitions  of  complex  systems,  integrated  teaming  is  not  an  option  but  a 
necessity.  Thus,  ACQ  has,  instead  of  an  addition  for  IPPD,  expected  material 
that  covers  integrated  teaming  derived  from  generalizing  and  simplifying  the 
IPPD  material  in  DEV. 

SVC  adopted  the  ACQ  approach,  but  in  many  service  contexts  “integrated  teams” 
were  not  the  key  differentiator  for  success  and  the  concept  also  proved  to  be 
problematic  in  some  contexts. 

Thus  to  harmonize  the  models,  a  different  approach  was  needed. 
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Teaming  Concepts2 


Overview  of  Solution 

Replaced  the  concepts  of  integrated  teaming  and  IPPD  with  a  more  general 
concept  of  teaming,  thereby  eliminating  the  IPPD  addition  and  making  the 
approach  to  teaming  consistent  in  all  three  models  (By  making  the  three 
constellations  common,  teaming  can  be  part  of  the  CMF.) 

Replaced  the  glossary  definition  of  “integrated  team”  with  a  definition  of  “team” 

In  the  glossary  definition,  placed  emphasis  on  what  enables  superior  team 
performance: 

A  team  establishes  and  maintains  a  process  that  identifies  roles, 

responsibilities,  and  interfaces:  is  sufficiently  precise  to  enable  the  team  to 

measure,  manage,  and  improve  their  work  performance:  and  enables  the 

team  to  make  and  defend  their  commitments. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cuml  1.3;  nov.  2010 

•  ©2010  Carnegie  Mellon  University 


Software  Engineering  Institute 


Carnegie  Mellon 


©  2006  Carnegie  Mellon  University 


4 


Presentation  Title 


10/29/2010 


The  Term  “Project”., 


The  Problem 

In  VI  .2  models,  the  word  “project”  was  used  in  all  three  CMMI  models, 
especially  in  the  core  process  areas.  “Project”  was  almost  implicitly 
understood  by  product  developers  and  acquirers. 

However,  service  providers  found  it  difficult  to  interpret  goals  and  practices 
containing  the  word,  often  misinterpreted  the  models  practices,  and 
sometimes  believed  that  model  content  containing  the  word  “project” 
did  not  apply  to  them. 

Although  some  users  (probably  familiar  with  development  environments) 
could  adjust,  others  had  great  difficulty  and  asked  many  questions. 


Software  Engineering  Institute  Carnegie  Mellon 
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Overview  of  Solution 

Kept  the  word  “project”  in  the  DEV  and  ACQ  models,  but  replaced  it  with  alternate 
terms  in  the  CMMI-SVC  model.  Depending  on  its  implied  meaning,  the  word 
“project”  was  generally  either  (1)  simply  removed,  (2)  replaced  with  the  word 
“work,”  or  (3)  replaced  with  the  word  “work  group.” 

These  changes  included  changes  to  process  area  names  (e.g.,  Project  Planning 
becomes  Work  Planning).  The  process  area  category  Project  Management  also 
became  Project  and  Work  Management. 

Although  the  wording  of  some  model  material  is  different  in  SVC  than  in  DEV  and 
ACQ,  if  the  only  difference  is  the  replacement  of  the  word  “project,”  the  material 
is  still  considered  CMF. 

Added  terms  and  revised  definitions  in  the  glossary  that  use  the  word  “project”  to 
ensure  that  the  glossary  more  broadly  fit  all  CMMI  models. 
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The  Term  “Project”3 


Example  -  Process  Area  Content 

Project  Planning  (ACQ  &  DEV) 

Purpose:  The  purpose  of  Project 
Planning  (PP)  is  to  establish  and 
maintain  plans  that  define  project 
activities. 


SG  2  Develop  a  Project  Plan 

A  project  plan  is  established  and 
maintained  as  the  basis  for  managing 
the  project . 


Work  Planning  (SVC) 

Purpose:  The  purpose  of  Work 
Planning  (WP)  is  to  establish  and 
maintain  plans  that  define  work 
activities. 


SG  2  Develop  a  Work  Plan 

A  work  plan  is  established  and 
maintained  as  the  basis  for 
managing  the  work. 
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Core  PAs:  Support  Category 


Configuration  Management 

establish  and  maintain  the  integrity  of  work  products 
using  configuration  identification,  configuration 
control,  configuration  status  accounting,  and 
configuration  audits 


<CM:  Clarified  that  CM  can  apply  to 
hardware,  equipment,  and  other 

tangible  assets. 


Decision  Analysis  and  Resolution 

analyze  possible  decisions  using  a  formal 
evaluation  process  that  evaluates  identified 
alternatives  against  established  criteria 

Measurement  and  Analysis 

develop  and  sustain  a  measurement  capability  used 
to  support  management  information  needs 

Process  and  Product  Quality  Assurance 

provide  staff  and  management  with  objective  insight 
into  processes  and  associated  work  products 


DAR:  Added  guidance  on 
preparing  to  use  DAR  practices  and 
communicating  results. 


MA:  More  clearly  distinguished 
between  information  needs  and 
objectives,  measurement 
objectives,  and  business/project 
objectives.  Included  a  table  of 
examples  (as  in  ACQ)  for  DEV 
and  SVC. 


< Clarified  that  PPQA  also  applies  to 
organization  level  activities  and 
work  products. 
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Core  PAs:  Process  Management  Category 


Organizational  Process  Definition 

establish  and  maintain  a  usable  set  of 
organizational  process  assets,  work  environment 
standards,  and  rules  and  guidelines  for  teams 


Organizational  Process  Focus 


plan,  implement,  and  deploy  organizational  process 
improvements  based  on  a  thorough  understanding 
of  current  strengths  and  weaknesses  of  the 
organization’s  processes  and  process  assets 


Organizational  Training 

develop  skills  and  knowledge  of  people  so  they  can 
perform  their  roles  effectively  and  efficiently 


<  Converted  goal  on  IPPD  or 
Integrated  Teaming  to  a  single 
practice  (IPPD  no  longer  an 
addition). 


< Expanded  applicability  to  training 
delivery  methods  such  as  self 
study,  mentoring,  and  online 
training. 
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Core  PAs:  Project  and  Work  Management 
Category  -1 


Integrated  Project  Management 

establish  and  manage  the  project  and  the 
involvement  of  relevant  stakeholders  according  to 
an  integrated  and  defined  process  that  is  tailored 
from  the  organization’s  set  of  standard  processes 

Project  Monitoring  and  Control 

provide  an  understanding  of  the  project’s  progress 
so  that  appropriate  corrective  actions  can  be  taken 
when  the  project’s  performance  deviates 
significantly  from  the  plan 

Project  Planning 

establish  and  maintain  plans  that  define  project 
activities 


Simplified  SP  1 .7  to  replace 
“work  products,  measures,  and 
documented  experiences”  with 

“process-related  experiences.” 

Converted  goal  on  IPPD  or 
Integrated  Teaming  to  a  single 
practice  (IPPD  no  longer  an 
addition). 


Added  guidance  for  monitoring  risks, 
data  management,  stakeholder 
involvement,  project  progress,  and 
milestone  reviews. 


,dded  guidance  on  determining 
project  lifecycle  and  milestones. 

Added  subpractices  on 
determining  data  rights  and 
version  control;  and  determining 
communication  and  other 
continuing  resource  needs. 
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Core  PAs:  Project  and  Work  Management 
Category  -2 


Requirements  Management 

manage  requirements  of  the  project’s  products  and 
product  components  and  to  identify  inconsistencies 
between  those  requirements  and  the  project’s  plans 
and  work  products 

Risk  Management 

identify  potential  problems  before  they  occur  so  that 
risk-handling  activities  can  be  planned  and  invoked 
as  needed  across  the  life  of  the  product  or  project  to 
mitigate  adverse  impacts  on  achieving  objectives 


Changed  the  focus  of  SP  1 .5  so  that 
it  now  reads  “Ensure  that  project 
plans  and  work  products  remain 
aligned  with  the  requirements.” 


icluded  examples  related  to: 
selected  architecture,  use  of 
industry  standards  to  identify 
risks,  FMEA,  and  consequence 
monetization. 

Provided  guidance  on 
laintaining  risk  parameters  thru 
life  of  the  project. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cumli-sjnov.  2010 

•  ©2010  Carnegie  Mellon  University 


SAM  -  the  Shared  PA 


SG  1 :  Establish  Supplier  Agreements 
SP  1 .1  Determine  Acquisition  Type 

SP  1 .2  Select  Suppliers 

SP  1 .3  Establish  Supplier  Agreements 

SG  2:  Satisfy  Supplier  Agreements 

SP  2.1  Execute  the  Supplier  Agreement 
SP  2.2  Accept  the  Acquired  Product 

SP2.3  Ensure  Transition  of  Products 


<  Clarified  the  applicability  of  SAM 

practices. 


Added  the  concept  “products  and 
processes  of  significant  value  to 
the  project”  to  help  determine  what 
to  monitor. 


Demoted  SP  2.2  and  SP  2.3  to 
subpractices  of  SP  2.1  and 
renumbered  the  remainder  of  the 
practices. 


Revised  SP  2.5  to  allow  its 
applicability  to  times  when  the 
product  or  service  is  delivered 
directly  to  the  customer  or  end 
user  from  the  supplier  (SVC  and 
DEV  only). 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Principles  of  Architecture-Centric  Practices 


1.  Regardless  of  scale,  architecture  is  the  appropriate  abstraction  for 
reasoning  about  business/mission  goal  satisfaction. 

2.  Quality  attributes  have  a  dominant  influence  on  a  system’s 
architecture. 

3.  Architectural  prescriptions  must  be  demonstrably  satisfied  by  the 

implementation. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 


Users  Need  Both  Functions  and  Qualities 


Required  capability 
Low  learning  threshold 
Ease  of  use 
Predictable  behavior 
Dependable  service 
Timely  response 
Timely  throughput 
Protection  from  unintended  intruders  and  viruses 


Software  system/mission  goals  should  address  user  needs. 

User  needs  often  translate  to  quality  attribute  requirements. 

Scenarios  are  a  powerful  way  to  characterize  quality  attributes  and 
represent  user  and  other  stakeholder  views. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Modern  Development  Practices  in  CMMI  - 1 


For  Version  1.3,  CMMI  provides  better  guidance  in  support  of  architecture¬ 
centric  practices 

•  creating  the  business  case  for  the  system  (partially  in  RD) 

•  understanding  the  requirements  (RD) 

•  creating  and/or  selecting  the  architecture  (TS) 

•  documenting  and  communicating  the  architecture  (RD,  TS) 

•  analyzing  or  evaluating  the  architecture  (RD,  TS,  VAL,  VER) 

•  implementing  the  system  based  on  the  architecture  (TS;  A/PL  notes) 

•  ensuring  that  the  implementation  conforms  to  the  architecture  (VER) 

•  evolving  the  architecture  so  that  it  continues  to  meet  business  and 
mission  goals  (implicit  in  the  phrase  “establish  and  maintain”) 


For  a  more  detailed  mapping  of  CM  Ml -DEV  VI  .3  to  the  above,  see  the  slides 
from  the  half-day  tutorial,  “CMMI  VI  .3  and  Architecture”  (session  11203). 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 


Modern  Development  Practices  in  CMMI  -  2 


CMMI  VI. 3  provides  improved  terminology  to  support 
architecture-centric  practices 

•  Updated  the  glossary  to  include  new  terms  (and  modified  some  old 
terms) 

•  Updated  the  informative  material  (especially  ARD  and  ATM  in  ACQ; 
RD,  TS,  and  VER  in  DEV;  and  SSD  in  SVC)  to: 

—  make  use  of  the  new  terms 

—  bring  more  emphasis  to  quality  attributes  and  thus  strike  a  better 
balance  between  functional  and  non-functional  requirements 

•  Replaced  selected  uses  of  overloaded  terms  such  as  “performance" 
with  an  appropriate  qualifying  phrase. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Modernizing  Development  Practices3 


Example  -  New  terms  reflecting  modern  engineering 
quality  attribute 

A  property  of  a  product  or  service  by  which  its  quality  will  be  judged 
by  relevant  stakeholders.  Quality  attributes  are  characterizable  by 
some  appropriate  measure. 

Quality  attributes  are  non-functional,  such  as  timeliness,  throughput, 
responsiveness,  security,  modifiability,  reliability,  and  usability.  They  have  a 
significant  influence  on  the  architecture. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 


Modernizing  Development  Practices4 


Example  -  New  terms  reflecting  modern  engineering 
architecture 

The  set  of  structures  needed  to  reason  about  a  product.  These 
structures  are  comprised  of  elements,  relations  among  them,  and 
properties  of  both. 

In  a  service  context,  the  architecture  is  often  applied  to  the  service  system. 

Note  that  functionality  is  only  one  aspect  of  the  product.  Quality  attributes,  such 
as  responsiveness,  reliability,  and  security,  are  also  important  to  reason  about. 
Structures  provide  the  means  for  highlighting  different  portions  of  the 
architecture.  (See  also  “functional  architecture.”) 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Requirements  Development 


SG  1 :  Develop  Customer  Requirements 
SP1.1  Elicit  Needs 

SP  1 .2  Transform  Stakeholder  Needs  into  Customer 
Requirements 

SG  2:  Develop  Product  Requirements 

SP  2.1  Establish  Product  and  Product  Component 

Requirements 

SP  2.2  Allocate  Product  Component  Requirements 

SP  2.3  Identify  Interface  Requirements 

SG  3:  Analyze  and  Validate  Requirements 

SP  3.1  Establish  Operational  Concepts  and  Scenarios 
SP  3.2  Establish  a  Definition  of  Required  Functionality 

and  Quality  Attributes 
SP  3.3  Analyze  Requirements 

SP  3.4  Analyze  Requirements  to  Achieve  Balance 

SP  3.5  Validate  Requirements 


<  Added  that  requirements  can  be 
monitored  through  development 
based  on  their  criticality  to  the 
customer  or  end  user. 


Revised  the  terminology  used  from 
a  strong  emphasis  on  “operational 
scenarios”  to  a  more  balanced 

“scenarios  (operational, 
sustainment,  and  development).” 


Added  “quality  attributes”  as 

properties  of  products  and  services 
in  addition  to  “functionality,”  which 
resulted  in  changes  to  SG3  and 
SP  3.2. 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cumli-sjnov.  2010 

•  ©2010  Carnegie  Mellon  University 


Product  Integration 


SG  1 :  Prepare  for  Product  Integration 

SP  1 .1  Establish  an  Integration  Strategy 

SP  1 .2  Establish  the  Product  Integration  Environment 

SP  1 .3  Establish  Product  Integration  Procedures  and 
Criteria 

SG  2:  Ensure  Interface  Compatibility 


Revised  the  purpose  statement  to 
ensure  proper  behavior  instead  of 
proper  function,  thereby  including 

quality  attributes. 


Changed  emphasis  on  integration 
sequence  to  an  emphasis  on 

integration  strategy. 


SP  2.1  Review  Interface  Descriptions  for 

Completeness 

SP  2.2  Manage  Interfaces 

SG  3:  Assemble  Product  Components  and  Deliver  the  Product 


Described  an  integration  strategy 
and  how  it  relates  to  an  integration 
sequence. 


SP  3.1  Confirm  Readiness  of  Product  Components  for 
Integration 

SP  3.2  Assemble  Product  Components 

SP  3.3  Evaluate  Assembled  Product  Components 

SP  3.4  Package  and  Deliver  the  Product  or  Product 
Component 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Addressing  Agile., 


The  Problem 

Developers  that  use  Agile  methods  sometimes  resist  using  CMMI  because  they 
can’t  see  how  CMMI  practices  can  complement  or  improve  the  effectiveness  of 
Agile  methods. 

Overview  of  Solution 

Added  guidance  to  the  appropriate  PAs  to  do  the  following: 

•  Help  users  interpret  the  practices  in  a  context  where  Agile  methods  are 
used 

•  Reinforce  the  applicability  of  the  practices  in  an  Agile  environment 

•  Send  the  message  that  CMMI  is  a  robust  best  practice  framework  meant  to 
be  used  in  Agile  environments  as  well  as  other  development  environments 


Software  Engineering  Institute  Carnegie  Mellon 


CMMI  VI. 3:  The  Rest  of  the  Story: 
Changes  at  CL/ML  1-3;  Nov.  2010 


Addressing  Agile2 


Solution 

Added  a  new  section  to  DEV  Chapter  5  entitled  “Interpreting  CMMI  When  Using 
Agile  Approaches” 

•  This  section  describes  how  CMMI  practices  can  apply  in  a  variety  of 
development  environments.  It  also  provides  interpretive  guidance  in 
selected  PAs  that  explains  how  the  PA  can  be  used  in  Agile  environments. 

•  A  reference  to  this  new  section  appears  in  the  SSD  intro  notes  of  SVC. 

Added  interpretive  guidance  to  the  following  PAs: 

•  In  DEV:  CM,  REQM,  PP,  RD,  TS,  PI,  VER,  PPQA,  and  RSKM 

•  In  ACQ:  AM,  ATM,  PMC,  and  PP 

•  In  SVC:  SSD 


Added  in  DEV  and  SVC  (SSD  only)  Agile-related  examples  (as  bullets) 


Software  Engineering  Institute  Carnegie  Mellon 


CMMI  VI. 3:  The  Rest  of  the  Story: 
Changes  at  CL/ML  1-3;  Nov.  2010 
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Addressing  Agile3 


An  example  of  a  note  added  to  DEV  is  the  following  one  for  PP: 

“In  Agile  environments  . . .  Teams  plan,  monitor,  and  adjust  plans 
during  each  iteration  as  often  as  it  takes  (e.g.,  daily).  Commitments 
to  plans  are  demonstrated  when  tasks  are  assigned  and  accepted 
during  iteration  planning,  user  stories  are  elaborated  or  estimated, 
and  iterations  are  populated  with  tasks  from  a  maintained  backlog  of 
work.  (See  “Interpreting  CMMI  When  Using  Agile  Approaches”  in 
Part  I.)” 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 


Summary  of  Global  Changes,  CMMI  Version  1.3 


Refining  CMMI  Framework  Related 
Terminology 

Updating  Model  Architecture 
Harmonizing  VI. 2  Models 
Glossary 

Teaming  Concepts 

The  Term  “Project” 

Process  Related  Experiences 

Providing  “Appropriate”  Phrasing  in 
Practice  Statements 

Generic  Practices 


Addressing  Agile 

Causal  Analysis  at  Lower  Levels  of 
Maturity 

Customer  Satisfaction 
Modernizing  Development  Practices 
Prioritized  Customer  Requirements 
Organization-Level  Contracts 
Easing  Translation 
Front  Matter 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 
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Contact  Information 


Mike  Konrad 
SEPM 

Telephone:  412-268-5813 
Email:  mdk@sei.cmu.edu 


U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 

Worldwide  Web: 

http://www.sei.cmu.edu/productlines 
SEI  Fax:  412-268-5758 


CMMIV1.3:  The  Rest  of  the  Story: 

^==-  Software  Engineering  Institute  Carnegie  Mellon  changes  at  cl/ml  i-s;  nov  .2010 

•  ©2010  Carnegie  Mellon  University 


Questions 


CMMIV1.3:  The  Rest  of  the  Story: 
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Benefits  of  CMMI 
Within  the  Defense 
Industry 


Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 


May  2010 


Software  Engineering  Institute 


Carnegie  Mellon 


Outline 


CMMI 


•  Introduction 

•  Benefits  of  CMMI  Implementation 

-Quantitative 

-Qualitative 

•  Perspectives  on  CMMI  Adoption 

•  Appraisal  Trends  -  2010 

•  Looking  Ahead 

•  Summary 


This  report  was  created  with  the  cooperation  of  the  Systems  Engineering 
Division  (SED)  of  the  National  Defense  Industrial  Association  (NDIA)  and 
their  member  companies  and  DoD  organizations. 


=•  Software  Engineering  Institute  Carnegie  Mellon 


Purpose  of  Presentation 


CMMI 


Present  new  evidence  about  effective  implementations  of  CMMI 

•  Examples  are  provided  by  the  defense  industrial  base  and  DoD  organizations. 

•  New  examples  are  based  upon  the  measures  that  practicing  organizations  use 
to  track  value  to  their  businesses. 

•  Examples  are  provided  by  organizations  that  have  tracked  and  measured 
performance  improvements  from  using  CMMI  over  many  years. 

•  Many  of  the  organizations  emphasize  high  maturity  results  and  show  that  they 
enabled  superior  performance. 

•  Their  data  indicate  why  CMMI  is  important  to  the  DoD  &  its  suppliers. 


The  new  data  presented  in  this  report  demonstrates  that  effective 
implementation  of  good  practices  aided  by  use  of  CMMI  can  improve 
cost,  schedule,  and  quality  performance. 


= -  Software  Engineering  Institute  Carnegie  Mellon 


CMMI:  Major  Benefits  to  DoD 

“Does  CMMI  work?”  We  asked  our  nation’s  defense  contractors,  as  well  as 
government  agencies,  to  share  results  from  their  performance  improvement  efforts 
using  CMMI.  The  results  spoke  for  themselves:  “Yes,  CMMI  works!” 


CMMI 


The  following  slides  include  information  from  six  defense  organizations  that 
responded.* 


Raytheon 


NORTHROP  GRUMMAN 


"Results  reported  in  this  presentation  are  not  attributed  to  protect  confidentiality. 
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Background  on  the  Data  for  this  €MMl, 

Presentation 

Organizational  and  project  leaders  decided  which  measures  were  most 
useful  to  them  when  tracking  the  results  of  CM  Ml -based  improvements. 

A  common  thread  was  their  interest  in  measuring  the  effect  CMMI  had  on 
schedule,  effort  and  cost,  and  quality. 

The  summarized  results  demonstrate  the  wide  scope  of  business  values 
and  goals  of  the  participating  organizations. 

The  source  studies  in  this  presentation  used  current  data  as  follows: 

•  2010:  Organizations  1, 2A,  3,  &  6 

•  2009:  Organizations  5  &  7 

•  2008:  Organization  2B 


Software  Engineering  Institute  Carnegie  Mellon 


Quantitative  Measures:  Schedule 
Performance  Results  Summary 


CMMI 


Measure  Used  By  The  Organization  Performance  Result 


On-time  deliverables  (Organization 
2a) 


Earlier  Defect  Detection  and  Repair 
(Organization  1) 


Schedule  performance  index 
(Organization  7) 


On-time  deliverable  improvement  of  4.9% 
out  of  a  possible  5%  improvement  - 
closed  the  gap  to  within  0.1%  of  perfection 
(organization  went  from  95%  to  99.9%  of 
deliveries  on  time) 

6.35  times  less  defect  discovery  and 
repair  hours  after  start  of  system  testing; 
potential  savings  of  5  -  6.5  months  in 
schedule  delay  after  system  tests  begin 
for  average  sized  project 

Increased  from  .78  to  .93  over  three  years 
(a  19.2%  improvement  in  estimation  and 
execution  of  schedule) 


Software  Engineering  Institute  Carnegie  Mellon 


Quantitative  Measures:  Effort  (Rework)  and 
Cost  Performance  Results  Summary 


CMMI 


Measure  Used  By  The  Organization 

Performance  Result 

Total  hours  for  defect  repair  (Organization  1) 

58%  fewer  hours  needed  to  repair  defects  for 
ML5  versus  ML3;  Result:  a  potential  cost 
savings  of  $1 .9  to  $2.3  M  per  average-sized 
project  (defined  as  233  KESLOC  [Kilo  Equivalent 
Source  Lines  of  Code]) 

Hours  per  KLOC  to  find  and  fix  defects  for  Defect  find  and  fix  cost  down  22% 
CMMI  ML5  relative  to  the  SW-CMMI  ML3 
baseline  (Organization  6) 


Effort  hours  needed  to  repair  high  severity  24%  reduction  in  effort  hours  per  defect 
defects  in  integration  and  test  phases 
(Organization  4) 

Cost  performance  index  (Organization  4)  Increased  from  .88  to  .96  over  two  years 


Overhead  rates  for  CMMI  ML5  relative  to  the  Reduced  by  7.3% 
SW-CMMI  ML3  baseline  (Organization  6) 


Software  development  cost  for  CMMI  ML5 
relative  to  the  SW-CMMI  ML3  baseline 
(Organization  6) 


_  Software  Engineering  Institute 


Reduced  by  28% 


Carnegie  Mellon 
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Selected  Results:  High  Maturity  Reduces 

Costs  for  Repair  (Organization  1) 


CMMI 
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57.7%  Fewer  Hours 
Overall  for  ML  5 


Req  &  Design  Code  &  UT  Sys  &  Acpt  Test  Post  Delivery  Total  Hours 


=■  Software  Engineering  Institute  Carnegie  Mellon 


High  Maturity  Projects 
Discover  defects 
earlier 


Early  detection  and  repair  lowers 
Costs 

57.7%  fewer  hours  for  ML5 
projects  expended  to  repair 
defects  versus  ML3 

105.3  fewer  hours  per  defect 

•  88.6  fewer  hours  during 
Testing  alone 

•  When  largest  risk  to 
schedule  occurs 
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Selected  Results:  Effort  to  Repair  Defects 

by  Phase  (Organization  1) 


CMMI. 


45,000 

40,000 
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Hours  to  Repair  Defects  (by  Phase) 
(233  KESLOC  Avg  Project) 


42,519 


I  Maturity  Level  3 
Maturity  Level  5 


57.7%  fewer  hours  (24,527) 
expended  for  ML  5 


6.35  times  (20,641  hrs)  less 
risk  of  Cost  or  Schedule 
impact  late  in  program 


24,496 


1,841 

5  2,177 

Req  &  Design  Code  &  UT  Sys  &  Acpt  Test  Post  Delivery  Total  Hours 


Potential  Cost  Savings  From  $  1.9  M  to  $2.3  M  per  average  sized  program 
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Quantitative  Measures: 

Quality  Performance  Results  Summary 


CMMI. 


Measure  Used  By  The  Organization 

Performance  Result 

Defect  density  by  severity,  ML5 
compared  to  ML3  (Organization  1) 

62.5%  fewer  high-severity  defects 
with  ML5  projects 

Defect  density  in  circuit  board  design 
(Organization  2a) 

65%  improvement 

Defect  containment  by  phase 
(Organization  3) 

The  fix  of  defects  within  the  phase 
they  were  injected  increased  by  240% 

Defect  containment,  ML5  compared  to 
ML3,  by  phase  per  KLOC  (thousands  of 
lines  of  code)  (Organization  2b) 

Defect  containment  improved  13% 

User  acceptance  test  defects  per  KLOC 
(Organization  7) 

Less  than  0.15  defects  per  KLOC 

%  of  defects  removed  prior  to  system 
test  (Organization  7) 

>85% 
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Selected  Results:  Quality  Performance 

(Organization  3) 


Defects  Phase  Containment  /  Leakage 
(High  Severity  Defects  - 
Priority  1,  2  &  3) 
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Within  ML5  projects: 

-  Defect  containment  (within 
phase)  is  increased  by  240 % 

-  Leakage  is  reduced  by  90%  for 
defects  discovered  “ 1  phase  later” 

84%  reduction  in  defects  leaked  “2 
or  2+  Phases' ' 

Optimizing  verification  activities: 

-  Peer  reviews 

-  Unit  testing 

-  Integration  testing 


CMMI. 


Captured  within  Leakage  =  1 

Phase 


Cost  avoidance  realized: 

-  Less  rework  late  in  the  life  cycle  when 
it  is  most  expensive  to  repair 

-  Resulting  in  reduced  schedule  risk 
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Quantitative  Measures: 
Productivity  Results  Summary 


CMMI 


Measure  Used  By  The  Organization 

Performance  Result 

Productivity  Gain  with  ML5  (Organization 

i) 

42%  gain  with  ML5  organizational 
practices  over  9  years 

Organizational  productivity  vs.  Galorath  Production  hours  reduction:  33.0% 
SEER  SEM  Estimation  Model  at  ML3;  37.4%  at  ML5 

(Organization  1) 

Productivity  for  CMMI  ML5  relative  to  the  Productivity  up  25.2% 

SW-CMM  ML3  baseline  (Organization  6) 
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Selected  Results:  Software  Productivity  cmmi 

(Organization  1) 


Productivity  Gain  with  Long  Term  ML  5 


Average  project  size  was  233 
KESLOC 


1.60 

1.40 

1.20 

1.00 

0.80 

0.60 

0.40 

0.20 

0.00 


42%  Gain  with  ML  5 


Largest  =  1 ,360  KESLOC 
Smallest  =  29  KESLOC 


9  Year  ML5  Org 


Other  Projects 


Average  customer  project  savings  due  to  increased  productivity 
•  Equivalent  of  406  work  months  per  project  (33.8  work  years) 


Software  Engineering  Institute  Carnegie  Mellon 


13 


Quantitative  Measures: 

Customer  Satisfaction  Results  Summary 


CMMI 


Measure  Used  By  The  Organization 


Performance  Result 


Award  fee  (used  as  an  indicator  of 
customer  satisfaction)  for  CMMI  ML5 
relative  to  the  SW-CMM  ML2  baseline 
(Organization  6) 

50%  of  potential  additional  award 
fee  achieved 

Cost  savings  to  customer  in  a  cost-plus 
contract  (Organization  1) 

Rose  from  5.7  M  to  7.1  M  (25%) 
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Selected  Results:  Award  Fee  (Organizations) 


CMMI. 


Potential 
Additional 
Award  Fee 
Available 


Percent 


50%  of  Potential 
Additional  Award  Fee 
Achieved 


SW  CMM  L2  SW  CMM  L3  SW  CMM  L4  SW  CMM  L5  CMMI  L5 


CMMI  L5 


Customer  Satisfaction  Continues  to  Improve 
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Quantitative  Result:  Return  on  Investment 

(Organization  2a) 


CMMI 


Organization  2a  reported  their  quantified  ROI  from  CMMI  Maturity  Level  5 
activity  to  be  24  :  1 . 


Using  the  data  in  Performance  Results  of  CMMI®  -Based 
Process  Improvement  (CMU/SEI-2006-TR-004)  they 
were  able  to  compare  their  ROI  performance  to  others  in 
industry: 


Median  ROI 
Lowest  ROI 

Organization  2a 
Highest  ROI 


4  :  1 

1.7  :  1 

24  :  1 

27.7  :  1 


These  results  are  a  consequence  of  meaningful  process  improvement  aligned  with 
the  business  and  engineering  objectives. 
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Putting  This  Presentation  In  Perspective  > 

CMMI 

The  reporting  organizations  have  a  long  history  of  commitment  to 
CMMI 


Many  have  a  history  of  multiple  reappraisals  at  high  maturity  levels  over 
many  years 

All  these  organizations  clearly  embody  an  approach  that  incorporates 
adherence  to  process  as  part  of  their  operating  principles 


None  of  these  organizations  would  have  been  able  to  collect  and  analyze 
the  years  of  data  needed  to  document  these  results  if  they  were 
“bookshelf  process”  organizations 
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Putting  This  Presentation  In  Perspective  -  2  cmmi 


Each  organization  measured  their  own  internal  processes 

•  CMMI  measures  how  well  an  organization  follows  their  process 

•  Maturity  Levels  are  used  to  provide  a  relative  basis  of  comparison 

•  CMMI  is  a  good  tool  to  define  states  of  maturity 

•  Measurements  demonstrate  the  value  received 

The  benefits  described  came  from  execution  of  each 
organization’s  processes 

•  Each  organization  identified  areas  different  areas  of  focus 

•  Gains  were  achieved  when  process  focus  occurred 


CMMI  Provides  Many  Qualitative 
Benefits  as  Well* 

Organizations  also  gathered  various  qualitative 
measures  to  compliment  their  quantitative 
measurements.  They  found  qualitative  benefits  such  as: 


CMMI< 


Reduced  overtime  and  less  intense 
pressure 

Clear  roles  and  responsibilities  for 
business  execution 

Common  language  (i.e.,  defined 
processes,  measures)  across 
business  units 

Decrease  in  replanning 

Products  with  lower  levels  of  defects 
and  lower  risk;  one  organization  offers 
a  lifetime  warranty  on  products 


Improved  program  insight, 
control,  and  tracking 

Reduced  training:  process 
documentation  enables  knowledge 
transfer  to  new  generation  of  workers 

Process  transformation  (via 
consistency,  integration,  coordination) 

Personnel  retention  and  job  satisfaction 


'based  on  published  benefits  from  a  wide 
variety  of  organizations 
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The  Bottom  Line 


CMMI 


Why  improve  processes?  -  Because  processes  are  the  foundation 
for  all  other  business  improvements,  and  critical  for 

•  lasting  improvements 

•  successful  technology  insertion 

If  a  performance  management  system  is  not  in  use,  leadership  is 
unaware  of  what  is  and  is  not  working. 

CMMI  is  a  proven  approach  to  performance  management  -  with 
more  than  a  decade  of  results  showing  it  does  work. 

Organizations  have  provided  data  that  shows  CMMI 

•  enables  the  delivery  of  lower-defect  products,  with  predictable 
cost,  schedule,  and  quality 

•  improves  business  performance 

•  serves  as  competitive  discriminator 


CMMI  Works  for  Organizations  of  All  Sizes 


501  to  1000 
6.8% 


1001  to  2000 
3.7% 


2000+ 

2.5% 


1  to  100 
58.2% 


301  to  500 
7.4% 


25  or  fewer 
13.6% 


CMMI 


201  to  300 
9.0% 


26  to  50 
15.6% 

201  to  2000+ 

23.8% 

101  to  200 
19.8% 


51  to  75 
12.8% 


76  to  100 
8.9% 


Source  for  these  statistical  analyses:  http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 
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CMMI  Adoption  Is  Multi-Sector 


Retail  Trade 
0.3% 


Business  Services 
35.0% 


Services 

71.1% 


Wholesale  Trade 
0.1% 


Transportation, 

Communication,  Electric,  Gas 
and  Sanitary  Services 
3.6% 


Health  Services 
1.3% 


Finance,  Insurance  and  Real 
Estate 
5.5% 


Public  Administration 
(Including  Defense) 
3.2% 


Fabricated  Metal  Products 
0.2% 


Primary  Metal  Industries 
0.3% 


Industrial  Machinery  And 
Equipment 
0.7% 


Electronic  &  Other  Electric 
Equipment 
10.4% 

Instruments  And  Related 
Products 
1.0% 

Transportation  Equipment 
2.4% 


Other  Manufacturing 
Industries 
1.2% 


CMMI. 


Engineering  &  Management 

Services  Other  Services 

24.2%  10.7% 


Manufacturing 

16.3% 


Source  for  these  statistical  analyses:  http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 
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CMMI  Adoption  Knows  No  Borders 

There  are  33  countries  with  more  than  ten  appraisals  as  of  March  2010: 


CMMI 


USA 

1582 

China 

1229 

India 

524 

Japan 

306 

Spain 

180 

France 

168 

Korea  (ROK) 

165 

Brazil 

144 

Taiwan 

134 

U.K. 

113 

Mexico 

86 

Argentina 

77 

Germany 

76 

Malaysia 

71 

Canada 

59 

Egypt 

43 

Italy 

43 

Thailand 

38 

Chile 

37 

Australia 

36 

An  estimated  1 .8  million  people  work  in  organizations 
that  have  had  at  least  one  SCAMPI  A  appraisal  since 
April  2002. 


Also:  Colombia,  Pakistan,  Philippines,  Singapore,  Israel,  Hong  Kong, 
Vietnam,  Turkey,  Netherlands,  Portugal,  Sri  Lanka,  Ireland  and  Russia 
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Countries  With  Current  Appraisals 


-  This  List  Might  Contain  Several  Surprises 


Distribution  of  Appraisals  by  Country  -  Top  20 


■  China 

■  USA 

■  India 

■  Spain 

■  Brazil 

■  Japan 

■  South  Korea 

■  France 

■  Taiwan 

■  Mexico 

■  Argentina 
United  Kingdom 

■  Malaysia 

■  Thailand 
Italy 

■  Columbia 
Germany 
Chile 
Canada 
Pakistan 
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Data  through 
Nov  12,  2010 
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Results  Depend  on  Implementation 


CMMI 


Simply  deciding  to  “do  CMMI”  is  not  enough  to  achieve  benefits. 


Defining  good  processes,  using  them,  measuring  the  results,  and 
making  improvements  based  on  what  you  learn  are  all  key  to  reaping 
the  benefits  described  in  this  presentation. 


The  CMMI  models  are  a  foundational  part  of  a  comprehensive  approach 
to  process  improvement  that  helps  organizations  understand 

•  why  they  should  improve 

•  what  frameworks  and  tools  would  best  fit  their  needs 

•  how  to  implement  them 
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What  Does  Adoption  Data  Tell  Us 


Growth  in  High  Maturity  appears  to  be  stagnated 
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CMMI 


Data  through 
Nov  12,  2010 
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Unmasking  the  HiMat  Trends 


* — -Level  4  - Levels 


CMMI 


Data  through 
Nov  12,  2010 


Real  Growth  in  ML  4  &  ML  5  over  5  years  -  recent  ML  4  tracks  the  ML  3  surge 
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Positive  Trend  in  Hi  Mat  since  2008 
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Data  through 
Nov  12,  2010 
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Misreading  the  Tea  Leaves  on  HiMAT? 


CMMI 


The  huge  surge  in  ML  2  &  3  appraisals  starting  in  2007  may  have 
created  a  masking  function  as  the  validity  limit  on  appraisal  life  was 
taking  effect 


•  Reductions  evident  in  ML  4  &  5  have  not  been  evident  for  ML  2  &  3 


Maturity  growth  normally  takes  several  years  to  occur 

•  SW  CMM  maturity  took  over  6  years  to  take  hold  so  the  condition  is 
not  new 


We  may  have  seen  the  HiMAT  %  reduced  simply  because  appraisal 
totals  were  growing  at  an  accelerated  rate 


We  need  to  monitor  these  trends. 

HiMAT  growth  may  have  been  more  stable  than  we  thought 
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Recent  Research  on  CMMI:  Just  the  Tip 
of  the  Iceberg! 


CMMI 
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Looking  Ahead 

The  road  ahead  for  CMMI  implementation 

•  A  continued  focus  on  high  maturity 

More  and  more  organizations  are  striving  for  and  achieving  high  maturity  -  and  are 
collecting  data  demonstrating  the  benefits.  Once  at  ML  4  or  5,  organizations  must  maintain 
their  focus  on  good  implementation  practices  for  continuous  improvement. 

•  Implementation  of  CMMI  for  Services  (CMMI-SVC) 

CMMI-SVC  extends  the  benefits  of  CMMI  to  a  new  audience.  Service  providers  can  use 
the  model  concept  that  has  proven  useful  in  the  development  community  to  specifically 
address  their  interests  and  concerns. 

•  Implementation  of  CMMI  for  Acquisition  (CMMI-ACQ) 

CMMI-ACQ  helps  organizations  improve  relationships  with  their  suppliers  and  improve 
acquisition  processes.  The  model  can  enable  increased  control  of  projects,  better 
management  of  global  sourcing  of  products  and  services,  and  more  successful  acquisition 
solutions. 

•  Integration  with  other  improvement  paradigms  (e.g.,  TSP,  ISO,  Lean  Six  Sigma) 

Organizations  are  finding  that  integrated  improvement  initiatives  can  produce  outstanding 
results.  Choosing  CMMI  doesn’t  mean  discontinuing  improvement  efforts  already  in  place 
or  avoiding  new  ones  that  show  promise. 


CMMI. 
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Summary  cmmi 

Many  stakeholders  are  involved  in  the  development  and  maintenance  of  CMMI 
models,  with  participants  from  commercial  industry,  government,  and  the  DoD. 

Broad  adoption  has  occurred  worldwide.  Adopters  range  from  small  and  midsize 
organizations  (these  are  the  majority)  to  large  and  very  large  organizations. 

Organizations  that  provide  products  and  services  to  the  DoD  use  CMMI  to  improve 
programs,  systems,  product  and  service  management,  systems  and  software 
engineering,  work  processes,  and  training  solutions. 

Quantitative  and  qualitative  results  have  been  documented  by  defense  contractors 
and  others,  as  shown  in  this  report.  There  is  a  great  deal  of  additional  data  showing 
the  benefits  of  CMMI  from  a  broad  range  of  industries,  including  banking  and 
finance,  manufacturing,  medical,  and  others. 

CMMI  enables  performance  improvement  focused  on  business  objectives,  but  the 
level  of  success  depends  on  the  implementation. 
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Who  Benefits  from  CMMI  Today? 


CMMI 
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A  Professional  Services  Technology  Firm 

•  Founded  in  1992,  building  technology  solutions  for  bank  and  & 
brokerage  clients 

•  Expanded  to  complex  transaction,  system  integration,  and  compliance 
problems 

•Solutions  used  by  over  200,000  professional  users  in  140  countries 
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•  Preparing  for  external  audits/appraisals 
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Overview 


Organizations  typically  leverage  multiple  models  to  guide  internal 
processes. 


Internal  Processes 


External  Models 


KINGLAN 

SYSTEM 


D 


The  trick  is  to  effectively  map  them  for  a 
complete  enterprise-wide  view. 


IX 


Our  Original  Approach  to  PPQA: 
Individual  Process  Audits 

•  Started  with  separate  audits  for  each  process 

•  Each  process  was  audited  one  time  per  project 

•  Averaged  5  audits  per  project,  at  ~4  hours  each 

•  Paper  audit  checklists  were  used 
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Our  Original  Approach  to  PPQA: 
Individual  Process  Audits 

•  Audits  focused  on  the  evidence  existing,  rather  than  the  quality 
of  the  evidence/effectiveness  of  the  process 

•  No  direct  traceability  back  to  the  CMMI  model 

•  This  model  was  unsustainable: 

•  Time  intensive,  manual 

•  More  audits  as  processes  were  added 

•  Didn’t  ensure  compliance  across  a  project’s  lifecycle 
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Our  Original  Approach  to  PPQA: 
Individual  Process  Audits 


flKINGLAN 

SYSTEM 


Internal  Processes  and  External  Models  were  silos, 
D  with  no  traceability  or  visibility  across 
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Overview  -  Internal  Perspective 


Overview  -  External  Perspective 


Overview  -  Combined  Perspective 


r£ 


Internal 

Processes 


CMMI 

Model 


PPQA 

r  *  , 

SCAMPI 

Audits 

Appraisals 
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The  goal  is  to  have  a  comprehensive  view  across 
internal  process  and  external  model  compliance 
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Second  Approach  to  PPQA: 

Milestone  Audits 

•  Performed  audits  across  processes  at  milestones: 

•  Planning  Complete 

•  Elaboration  Complete 

•  Development  Complete 

•  Transition  Complete 

•  Examined  aspects  of  the  processes  that  were  appropriate  for 
each  milestone 

•  Ensured  compliance  across  a  project’s  lifecycle 

•  Began  to  leverage  tooling  to  eliminate  manual  effort 
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Second  Approach  to  PPQA: 
Milestone  Audits 

•  Began  using  tooling  to  support  Audits 


•  Audited  by  process  step,  focusing  more  on  quality  of  evidence 
and  effectiveness  of  the  process 


Element  Name 

Q  -|  K5C  PPQA  v  1.7 

a 

4 . 

PDS 

► 

PS  !  1  ■  2  Establish  Customer  Understanding  Expectations 

PS  1.3  Plan  for  Requirements  Changes 

PS 

1.4  Finalize  Requirements  Management  Planning 

. PDS  2.0  Establish  an  Initial  Requirements  Baseline 

PS 

2.1  Determine  Requirements  Needs 

PS  2.2  Propose  Initial  Requirements  Baseline 

PS 

2.3  Achieve  Initial  Customer  Understanding 

l .  PS  2.4  Establish  Initial  Requirements  Baseline 

4 . 

: 

PDS  3-0  Maintain  a  Requirements  Baseline 

-1 

k  . 

^  1  Co^^fce  Re^^ents^^^ine 

Process 
Goal  Level 
Process  Steps 
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Second  Approach  to  PPQA: 


Started  to  see  internal  process  traceability,  but  still  no 
visibility  across  to  the  external  models 


pjKINGLAND 

^40  SYSTEMS 


13 


Current  Approach  to  PPQA 

•  Still  perform  audits  across  processes  at  milestones 

•  Still  examine  process  sequences  that  are  appropriate  for  each 
milestone 

•  Still  ensure  compliance  across  a  project’s  lifecycle 

•  Increased  use  of  tooling  to: 

•  Support  Audits 

•  Report  compliance  against  internal  processes 

•  Map  and  identify  gaps  across  external  models 

•  Different  Approach  to  audit  focus 
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Current  Approach  to  PPQA 


•  Audit  projects  at  the  goal  level,  ensuring  that  projects: 

•  Meet  the  goal  of  each  process  sequence 

•  Their  plans  are  consistent  with  the  process 

•  Their  activities  are  consistent  with  their  plans 


KINGLAND 


Element  Name 


Q  K5C  PPQA  v  1.7 


0  ep  Requirements  Management 


0- 


PDS  10  Plan  for  Requirements  Management 


PS  1.1  Assign  Resources 


PS  ]i  .2  Establish  Customer  Understanding  Expectations 


PS  13  Plan  for  Requirements  Changes 


PS  14  Finalize  Requirements  Management  Planning 


PDS  2.0  Establish  an  Initial  Requirements  Baseline 


PS  2.1  Determine  Requirements  Needs 


PS  2.2  Propose  Initial  Requirements  Baseline 


PS  2.3  Achieve  Initial  Customer  Understanding 


PS  2.4  Establish  Initial  Requirements  Baseline 


PDS  3.0  Maintain  a  Requirements  Baseline 


.  1  Co^^fce  Re^^aents^^jjne 


Goal  Level 


Guiding  Principles 

•  All  processes  will  be  audited 

•  Ensure  everyone  is  performing  consistently 

•  Compliance  evaluated  against  goals  of  each  process  sequence 

•  Map  internal  processes  to  external  models,  driving  the  model  to 
the  background 

•  Leverage  audits  to  provide  coaching  as  well  as  provide  a  vehicle 
for  process  improvement 

•  Provide  feedback  to  the  project  teams  and  follow-up  on  non- 
compliance  items 
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Lessons  Learned 

•  Evaluation  at  the  goal  level  allows  for  better  mapping  to  external 
models 

•  Need  objective  rules  in  place  forevaluation 

•  Goal  Achieved 

•  Team’s  Plan  consistent  with  the  Process 

•  Team’s  Activities  consistent  with  their  plan 

•  Auditor  Guidance  within  checklists  is  a  must  to  maintain 
consistency  of  audits 

•  Having  auditors  review  evidence  in  advance,  followed  by 
interviews  with  team  members  is  much  more  efficient  (SCAMPI- 
like  events) 


l&KIN  GLAND 
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Lessons  Learned 

•  Tooling  allowed  us  to  have  better  record  retention,  trending,  and 
reference  for  internal  audits 


•  Reduced  average  time  spent  on  audits  per  project  by  24% 


•  Increased  process  compliance  by  46% 


4feKIN  GLAND 
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Current  Approach  to  PPQA 


iso 

27001 


Project 

Mgmt 


The  current  approach  allows  us  to  have  a 
comprehensive  view  across  internal  processes  and 
$;K  I  N  G  L  A  N  D  external  models 

5  V  5  T  E  M  S 
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Tooling 

Appraisal  Wizard  Suite  by  Integrated  System  Diagnostics,  Inc. 

•  Appraisal  Wizard 

•  Used  for  PPQA  Audits  and  SCAMPI  Appraisals 

•  Database  of  checklist  questions,  evidence  examined,  and  ratings 
tied  to  the  process(es)  being  evaluated 

•  Model  Wizard 

•  Used  to  design  models  for  use  in  Audits  and  Appraisals 

•  Ability  to  import  organizational  processes 

•  Contains  models  such  as  CMMI,  ISO,  etc. 

•  Model  Mapper 

•  Used  to  map  internal  processes  to  external  models 

•  Contains  maps  across  external  models  such  as  CMMI  to  ISO,  etc. 


tfeKIN  GLAND 
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Appraisal  Wizard 


Options  Tree  View  Record  View  ,  i=.n  i; 

PPOIB  Jft  lit-  v 

Model  KSC_PPQA_vl,7.  v  Element  Type  [ALL]  v  „ 

Rating  Level  Compliance  Solutions 

v  3=  Element  Color:  Process  Step  v  v 

Record  Filter  NONE  v 

J  ^ 

Name 

j  KSC  PPQA  vl,7 

gp  Requirements  Management 
r-i  PDS  1.0  Plan  for  Requirements  Management 

PS  I  1  ftisinn  Rpviiirm 


PS  ;  1 . 2  Establish  Customer  Understanding  Expectations 


PS  1.3  Plan  for  Requirements  Changes 
PS  1 . 4  Finalize  Requirements  Management  Planning 
-  P  DS  2.0  Establish  an  Initial  Requirements  Baseline 
PS  2.1  Determine  Requirements  Needs 
PS  2.2  Propose  Initial  Requirements 
PS  2.3  Achieve  Initiahjj^m^j^r^hding 
PS  2.4  Est^kh  ftal^qS^rnents  Baseline 
P  D  ^0^lain«n  ^^^birants  Baseline 

®mplete  Requiremenlj^St n^ 

PS  3.3  Estal^^^jr^ents  Baseline 
[_i  PDS  4.0  Manage  Requirements  Change  and  Inconsistency 
PS  4.1  Review  Requirements  Change  Requests 
PS  4.2  Monitor  for  Requirements  Inconsistencies 
Rl.l)  Perform  Specific  Practices 
R2.1)  Establish  an  Organizational  Policy 
R2.2)  Plan  the  Process 
R2.3)  Provide  Resources 
R2.4)  Assign  Responsibility 
R2.5)  Train  People 


Jj* 

rathe 


’rocess  Goal.  T o  develop  a  plan  that  the  team  will  use  to  identify  approved  requirements,  maintain  the  inte 
of  the  requirements  and  related  work  products  by  identifying  and  managing  inconsistencies,  and  managing 
changes  to  requirements. 

Auditor  Guidance.  The  REQM  plan  used  by  the  team  must  specifically  identify  the  sources  of  requirements, 
who  is  authorized  to  approve  requirements  and  changes,  how  the  requirements  ate  base  lined  and  controlled, 
and  the  manner  in  which  these  approvals  and  changes  will  be  will  be  managed  The  plan  must  also  identify  how 
requirements  related  work  products  will  be  monitored  for  inconsistencies  and  how  these  inconsistencies  will  be 
managed  to  closure.  Examples  of  inconsistencies  include  such  things  as  work  being  performed  that  is  outside 
the  scope  of  the  requirements,  or  identified  requirements  with  no  associated  planned  work  activities  or  simply  not 
addressed  within  the  developed  work  products.  Monitoring  for  inconsistencies  is  a  key  aspect  of  scope  control. 

Evidence.  Ensure  that  the  project  team  has  an  Approved  REQM  plan  that  clearly  outlines  all  aspects  of 
Requirements  Management  in  accordance  with  the  Registered  Process.  Also  test  to  ensure  that  the  project 
team  actually  understands  and  is  using  the  plan  that  they  have  developed. 

•REQM  Plan 

■  Affirmation  evidence  that  the  team  understands  and  is  using  the  REQM  Plan 


Record  Text  GLBL 

Describe  how  you  developed  and  communicated  your  plan  for  identifying,  elaborating  and  managing 


.  requirements, 
contained  within  REQM  Plan 


n  Summary  Rpts  Rec  Date  /  Time  ’ 

8/25/2010  10:48:23  1 


£/ 


eme 


nt 


Does  the  REQM  plan  address  the  following  areas? 

Acceptable  sources  to  provide  requirements  to  the  team? 

Acceptable  forms  of  requirements  and  how  they  will  be  approved? 

-  Means  for  which  the  team  will  ensure  a  common  understanding  of  requirements  between  the 
customer  and  the  team? 

-  How  changes  to  requirements  will  be  managed? 

-  How  requirements  baselines  will  be  established  and  maintained? 

-  How  commitments  to  individual  requirements  will  be  managed? 

-  How  inconsistencies  between  the  requirements  baseline  and  work  activities  and  related  work 
products  will  be  monitored  for,  and  how  discovered  inconsistencies  will  be  addressed? 

-  How  bi-directional  traceability  between  requirements  and  work  products  will  be  managed? 


0041  j  REQM  Plan  vl.Q 
0057  Initial  Enhancements  Lists 
^9  Project  Summaries 


http :  //kinaland/sites/comoliance  solution/indv%206  0/v60%20Pro1ect%20Administr 

http://kinqland/s.ites/cgmpliance  splutionjindy%206  p/y60%2.0Proiect.%20Administr 

http;//kinqland/sites/compliance  solution/indy%206  0/v60%20Requirement%20Defi 


Record  Documents 


-  BBS  -  Q*l-  6) 
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Appraisal  Wizard  -  Element  Name 


Element  Name 

_  -  -  K5CPPQA  vl.7 

B  RP  Requirements  Management 

1.0  Plan  for  Requirements  Management 
PS  1.1  Assign  Resources 


PDS 


PS 


1.2  Establish  Customer  Understanding  Expectations 
PS  13  Plan  for  Requirements  Changes 
PS  14  Finalize  Requirements  Management  Planning 


0- 


PDS  2.0  Establish  an  Initial  Requirements  Baseline 
2.1  Determine  Requirements  Needs 


PS 


PS  2.2  Propose  Initial  Requirements  Baseline 
PS  2.3  Achieve  Initial  Customer  Understanding 


PS  24  Establish  Initial  Requirements  Baseline 


PDS  3.0  Maintain  a  Requirements  Baseline 


P^^.l  Re^p»ents^^|ine 


feiiKINGLAND 
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Includes  the  model  you  are  auditing  against 
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Appraisal  Wizard  -  Element  Records 


:| 

Ret 

Record  Type 

Status 

Verification 

Record  Text 

3026 

Audit  Question 

Completed 

Yes 

Does  the  REQM  plan  address  the  following  areas? 

-  Acceptable  sources  to  provide  requirements  to  the  team? 

-  Acceptable  forms  of  requirements  and  how  they  will  be  approved? 

-  Means  for  which  the  team  will  ensure  a  common  understanding  of 
requirements  between  the  customer  and  the  team? 

-  How  changes  to  requirements  will  be  managed? 

-  How  requirements  baselines  will  be  established  and  maintained? 

-  How  commitments  to  individual  requirements  will  be  managed? 

-  How  inconsistencies  between  the  requirements  baseline  and  work  activities 
and  related  work  products  will  be  monitored  for,  and  how  discovered 
inconsistencies  will  be  addressed? 

-  How  bi-directional  traceability  between  requirements  and  work  products  will 
be  managed? 

Includes  questions  and  checklist  information  for  auditors 


flKINGLAN 

SYSTEM 
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Appraisal  Wizard  -  Record  Documents 


Di 

0041 

Title 

REQM  Plan  v  1.0 

File  Name  or  URL 

http :  //kJnciland/sites/compliance  solution/ind  y%206  0/y  60%20Proiect%20  Administr 

Evidence  Comments 

0052 

Requirements  Traceability  Matrix  vl.2 

http :  //kJnaland .  kJnaland .  cc/ sites/ compliance  solution/Document%20Library 6/ 1  /Indep 

Includes  evidence  examined  for  each  question 
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..  Ill 


Model  Wizard 


1 


Element  Name 

K5C  PPQA  vl.7 


EP 


EP 


Requirements  Management 
Supplier  Agreement  Management 


0  EP  Project  Initiation 

PDS  1.0  Idea  Generation 
0  PDS  2.0  Determine  Project  Vision 

PDS  3.0  Develop  Preliminary  Scope  Statement 
PDS  4.0  Analyze  Project  Feasibility 


I 

li¬ 


ps 


PS 


PS 


4.1  Determine  Technical  Feasibility 

4.2  Determine  Processes 

4.3  Determine  High  Level  Resources  &  Cost 


/ 

/ 

r 

4 


PS  -*■**  Determine  High  Level  Timeframes 
4.5  Determine  Project  Risk  Tolerance 


PS 


S- 


PDS  5.0  Complete  Project  Charter 

PS  5.1  Determine  Transition  and  Operations  Expectations  ^ 
PS  5.2  Determine  Project  Team  Leadership 
5.3  Conduct  Gap  Analysis 


/ 

4 

* 

4 

4 


PS 


0 . 


PDS 


PS  5.4  Complete  Project  Charter 
6.0  Obtain  Approval 

PS  6.1  Determine  Necessary  Approval  Level 
6.2  Review  Documents 


PS 


PS  6.3  Corrective  Action 


t 

. 

/ 


The  model  is  essentially  a  hierarchy  of 
the  processes,  sequences,  and  steps 
that  make  up  those  processes 


Staffing 

Assignment 

Process 


4.0 

Analyze 

Project 

Feasibility 


5.1 

Determine 
Transition  and 
Operations 
Expectations 


5.2 

Determine  Project 
Team  Leadership 


5.3 

Conduct  Gap 
Analysis 


Project  Charter 


Figure  5-  Complete  Project  Charter 

5.1.  Determine  Transition  and  Operations  Expectations 


6,0 

Obtain 

Approval 


5.1.1 


y  y  y  y  / 

tIKINGLAND 

^40^  SYSTEMS 


Step 


Analyze  the  project  to  determine  whether  the  project  will  create 
Transition  and  Operations  expectations.  In  the  event  that  the  project 
does  not,  please  state.  The  analysis  has  been  completed,  and  no 


Resource/Evi  den  ce 


Project  Charter 

//// 
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Model  Mapper 


Map  Elements  (MMGG7)  -  KristiTest 


Source  Model  Target  Model  Map 


/ 


H 


1 .0  Plan  for  Requirements  Management 


iZ  KSCPPQAvl.7 
-  EP  Requirements  Management 

Q  PDS 

PS  1 , 1  Assign  Resources 
PS  12  Establish  Customer  Understanding  Expecta 
PS  13  Plan  for  Requirements  Changes 
PS  1-4  Finalize  Requirements  Management  Planning 
0  PDS  2  0  Establish  an  Initial  Requirements  Baseline 
PS  2.1  Determine  Requirements  Needs 
PS  2.2  Propose  Initial  Requirements  Baseline 
PS  2.3  Achieve  Initial  Customer  Understanding 
PS  2.4  Establish  Initial  Requirements  Baseline 


;l  Element 

.1  CMMI-DEVj  VI. 2  Staged 


l 


Maturity  Level  2 

@  Requirements  Management 


El  (G)  REQM  5G  1 


i=i  (G)  REQM  GG  2 


REQM  GP2.1 
REQM  GP  2.2 


□  @REQM 


GP  2,3 


/ 


► 

►  a@ 

REQM  SR  1 . 1 

REQM  SR  1.2 

REQM  SR  1.3 

REQM  SR  1.4 

N 

REQM  SR  1.5 

7 


/ 


Enables  mapping  from  one  process  or  model  to 
another 


26 
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Model  Mapper 


Element 


2  JO  Establish  an 
Initial  Requirements 
Baseline: 


KSC  Process 


Mapped  Elements_ Link  Comment 


Link  Go m m ents : After  contingent  com  rmitm  ent  from  the 
team,  the  initial  requirements  are  provided  to  the 
customer  for  validation  and  approval.  Once  recieved, 
these  are  baselined  in  the  RTM  and  become  the  basis  for 
subsequent  planning 

CMMI  Model 


-  REQMSPl.l 


Certainty 


100 


Rationale  behind  the  mappings  and  coverage  certainty 
can  also  be  captured  and  used  for  additional  visibility 
and  reporting 


#&KIN  GLAND 
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Reporting  Compliance 


Generate  reports  on  internal  process  compliance 
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Reporting  Compliance 


■■ 


Gap  Analysis 


Report  -  Unmapped  Model  Elements 

KS C_PP Q Av  1 . 7  to  CMMI  Dev  vl.2 


Source  Model:  KSC  PPQA  vl.7 

Tniget  Model:  CMMI-DEV,  Vl.2  Staged 

KSC  PPQA  vl.7 

KSC  PPQA  vl.7 

Element  Name 

Element  Name 

KSC  PPQA  vl.7 

2.2  Establish  Evaluation  Criteria 

Requirements  Management 

2.3  Determine  List  of  Potential  Suppliers 

1.3  Plan  for  Requirements  Changes 

2.4  Gather  Inf onnation 

f  ^  E^“  -te r  ^  ^  ^  ^ 

Model  Map  Report 


KSC  PPQAvl.7  to  CMRH  Dev  vl.2 


Source  Model: 

KSC  PPQA  vl.7 

Target  Model: 

CMMI-DEV,  Vl.2  Staged 

Element 

Mapped  Elements 

Link  Comment 

Certainty 

1.0  Analyze 

Stakeholder  Needs: 

-  PP  SP  2.6 

Link  Comments: I identifies  stakholders  and  determines 
level  of  necessary  invovlement 

50 

-  PP  GP  2.7 

50 

////  ///  //////  /  /  ✓ 

30 
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SYSTEMS 


Gap  Analysis 


Create  reports  to  understand  alignment  of  internal 
processes  relative  to  external  models,  to  see  where 
iKINGLAND  there  are  gaps  and  strong  alignment 
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Preparing  for  SCAMPI  Appraisals 

•  From  our  internal  audits,  the  tool  already  includes 
evidence/information  in  preparation  for  SCAMPI  Appraisals 

•  Reduced  time  spent  in  preparation  for  a  SCAMPI  by  21% 


KINGLAND 


l lernem  Review  IAM009)  [lenient:  REQM  SP  5  1 


Options  Record  View  , 


I  Document  Filtc-rl 


Rating  Level  XYZ  Corp 


•  Record  Filter  NONE 


•" 


Model  Co  F 
OU  Cover  F 
Practice  I  Fully  Implemented 


Element  Records  Element  Documents  Document  List 


Drag  a  column  header  here 


1  Rec 

Record  Type 

Status 

Verification 

Record  Text 

GLBL 

Include  in  Sum  Rec  Date  /  Time 

1887 

JVPD  Project  PHD 

OE  Examined 

Yes 

How  do  you  develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements? 

2/24/2006  6:55:55 

PM 

The  business  requirements  were  developed  and  documented  by  interacting  with  the 
business  representative.  The  requirement  provider  is  identified  early  in  the  project 
life  cycle  and  they  enable  the  project  and  supplier  personnel  to  collect,  understand 
and  prioritize  the  business  requirements.  The  business  requirements  document 

□ 

l— 

□ 

i8i3: 

ISKD  Project  PIID 

OE  Examined 

Yes 

How  do  you  develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements? 

2/24/2006  6:55:37 

PM 

Project  developed  the  business  requirements  document  by  interacting  with  the 
business  representative.  The  business  representative  enables  the  project  team  to 
collect,  understand  and  prioritize  the  business  requirements.  The  business 
requirements  document  is  reviewed  and  approved  by  the  business  owner. 

□ 

□ 

1733 

ICVM  Project  PIID 

OE  Examined 

Yes 

How  do  you  develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements? 

9/18/2006  10:35:09 
AM 

The  SOW  document  which  details  the  requirements  is  signed  off  between  the 
business  representative  and  the  project  team  (includes  supplier).  The  records  of  the 
discussion  during  the  sign  off  are  maintained  as  minutes  of  meeting.  The  purchase 
order  is  released  as  evidence  of  acceptance  of  SOW  document 

□ 

□ 

E| 

GA  Project  PIID 

OE  Examined 

Yes 

How  do  you  develop  an  understanding  with  the  requirements  providers  on  the 
meaning  of  the  requirements? 

9/18/2006  10:34:38 

► 

1 

The  SOW  document  details  all  the  requirements  and  is  agreed  off  between  the 
business  representative  and  the  project  team.  The  business  representative  enables 
the  project  team  to  collect,  understand  and  prioritize  the  business  requirements. 

■ 

■ 

841 

Compliant/Normative 

Accepted 

Yes 

Planning  Managers  determine  the  high  level  requirements  and  then  assign  the  PM 
who  engages  the  vendor  and  business  representative  to  develop  the  BRD  and  SOW. 

□ 

□ 

9/18/2006  10:09:41 
AM 

Record  Fields  Record  Documents 


|  |®)  Create  New  Document  | 

by  that  column 

S  Di  Title 

File  Name  or  URL 

Evidence  Comments 

Evidence  Type  Doc  Status 

Doc 

1328  Statement  of  Work 

C.:\GA\L352  GA  SOW  1.2.doc 

Section  1 1 

Direct 

Plan 

1337  Business  Requirements  Document 

GlSAlL3i2_GA  JuanessReflujrementsDpcument.  1  .O^joc 

- j- 

Direct 

Plan 

1338  SOW  Approval  Meeting  Minutes 

C:\GA1L352  GA  SOW  ADoroval  Minutes  20040806.doc 

Indirect 

Sped 

32 


Conclusion 

How  do  you  ensure  that  your  program  has  a  complete  view  across 
internal  processes  and  external  models? 

•  Ensure  internal  processes  are  addressing  external  practices 

•  Map  internal  processes  to  industry  models 

•  Provide  a  means  to  report  on  and  modify  the  mapping 

•  Provide  visibility  of  compliance  between  internal  processes  and 
external  models 


#&KIN  GLAND 
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Questions  /  Discussion 


Thank  You 


For  questions  related  to  this  presentation,  please  contact: 


Kristi  Kingland  Glowaski 
641.355.1017 

Kristi.Kingland@Kingland.com 


SEI  Partner 


DILIGENT 

Consulting  Inc. 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Achieving  Successful 
CMMI-DEV  Practice  Implementation 
Across  Diverse  Distributed  Environments 

In  Small  Organizations 


CMMI 


SM  CMM,  CMMI,  SCAMPI,  SCAMPI  Lead  Assessor,  SCAMPI  Lead  Appraiser,  and  SEI  are  service  marks  of 
Carnegie  Mellon  University. 

®  CMM  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon 
university.  Diligent  Consulting  Inc. 


1 


SEI  Partner 


Agenda 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Agenda 

❖  Process  Improvement  Key  Factors  &  Challenges  for  Distributed  Small 
Businesses 

❖  Scenario  for  Implementing  Process  Improvement  Practices  Across 
Distributed  Small  Businesses 

❖  Bridging  Distributed  Environments  Using  CMMI-DEV  for  Distributed  Small 
Businesses 


❖  CMMI-DEV  Practice  Implementation  for  Distributed  Small  Businesses 

❖  Steps  for  CMMI-DEV  Practice  Implementation  for  Distributed  Small 
Businesses 


Diligent  Consulting  Inc. 
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SEI  Partner 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


diverse 

applications 


limited 

resources 


dynamic 

requirements 


customer 

types 


aggressive 

schedules 


CMMI 


\  J  Challenges 

Dynamic 

Requirements 

Impact  of  customer  co-location  on 
requirements  management  &  tasking 

Aggressive 

Schedules 

Immediate  effect  of  changing 
customer  priorities  &  direction 

Limited 

Resources 

Typical  work  performed  on  a  fixed 

LOE  or  time  &  material  basis. 

Diverse 

Applications 

Lack  of  institutionalized  project 
management  procedures,  templates 
and  project  past  performance  data. 

Customer 

Types 

Customers  have  different  cultures  & 
performance  expectation^ _ 

Diligent  Consulting  Inc . 


Scenario  for  Implementing  Process 
Improvement  Practices  Across  Distributed 
SEI Partner  Small  Businesses 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Distributed  small  businesses  often  function  in  focused 
markets  as: 

•  Suppliers  of  specialized  on-site  technical  services , 
key  personnel  or  products 

•  Domain  specialists 

•  Staff  augmentation 

These  focused  markets  may  not  provide  needed 
revenue  or  market  share  growth  to  readily  prosper  in 


today’s  economy. 


Costs  of  improving  management  practices  are  considered 
key  investments  in  these  vulnerable  cash  flow 
environments. 

Both  near  and  long  term  cash  flow  analysis  typically 
includes  factors  such  as: 

•  Customer  base  stability  &  revenue  cycles 

•  Projected  revenue  growth 

•  Costs  of  acquiring  appropriate  resources  to  improve 
management  practices,  such  as  training,  process 
engineers  and  collaborative  software  and  hardware. 


CMMI 


Diligent  Consulting  Inc. 
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Scenario  for  Implementing  Process 
Improvement  Practices  for  Distributed  Small 
SEI Partner  Businesses  (continued) 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Understanding  and  improving  project  management 
practices  is  a  key  factor  in  revenue  growth. 

Planning  improvement  of  project  management  typically 
includes  assessment  of  critical  factors  such  as: 

•  Required  resources  (available  staff,  feasible  schedule, 
platforms,  facilities,  collaborative  tools) 

•  Past  performance  data  (costs,  schedules,  lessons 
learned) 

•  Training  in  the  domain 

•  Risk  identification  and  impact  assessment 

•  Risk  mitigation 


Process  models  such  as  CMMI  and  ISO  9000  provide 
practical  knowledge  bases  for  improving  management 
practices. 

Obtaining  CMMI  benchmarks  and  ISO  9000  certifications  are 
often  stepping  stones  to  expand  government  and  commercial 
customers. 

Selecting  and  implementing  any  process  models  is  both 
costly  and  can  be  risky  as  there  are  no  guarantees  of  success 
in  distributed  customer  sites. 


CMMI 


Diligent  Consulting  Inc. 
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Scenario  for  Implementing  Process 
Improvement  Practices  for  Distributed 
SEI Partner  Small  Businesses  (continued) 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Planning  a  process  improvement  strategy  typically 
includes  identifying  problematic  projects  with  typical 
issues  such  as: 

•  Inaccurate  planning  data  for  costs  and  schedule 

•  Limited  resources  and  ambitious  schedules 

•  Lack  of  experience  and  staff  training 

•  Excessive  staff  turnover  rate 

•  Stability  of  current  customer  base 

•  Projection  of  market  growth 

•  Over  commitment  of  key  staff 


Implementing  a  process  improvement  strategy, 
distributed  businesses  should  customize  a  path  of 
small,  adjustable  steps. 

An  example  of  a  path  of  small  steps  is  a  hybrid 
approach  of  improving  project  management  practices 
in  CMMI-DEV  first. 


The  CMMI  project  management  practices  can  provide  a 
framework  for  future  adoption  of  ISO  9000. 


CMMI 


Diligent  Consulting  Inc. 
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Bridging  Distributed  Environments  Using 
—  CMMI-DEV  for  Distributed  Small 
SEI Partner  Businesses 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


CMMI-DEV 

Provides 

Commonality 


Customer  focused 
Requires  continuous  improvement 
Built  on  processes 
Focuses  on  resource  management 
Based  on  measurement  and 
analysis 


Bridging 
Techniques  to 
Achieve 

Institutionalization 


CMMI 


•  Provide  shared  remote  access 

•  Conduct  site  visits  to  understand 
different  environments  &  cultures 

•  Take  the  initiative  to  talk  daily 

•  Schedule  weekly  telecons  with 
individuals  at  each  site 


Integrate  technical  and  engineering 
activities  with  life-cycle  development 
tools  that  adapt  to  the  dynamics  of 
customer  environments 


Diligent  Consulting  Inc . 


CMMI-DEV  Practice  Implementation  for 
SEI Partner  Distributed  Small  Businesses 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Frameworks  of  the  CMMI  models 
provide: 

•  Increases  in  implementation 
efficiency  due  to  redundancy  in 
model  functions,  such  as  Project 
Management  &  Support  Process 
Areas  and  generic  practices. 

•  Minimal  opportunities  for 
conflicts  with  staff  and  resources 
with  adequate  model 
understanding 


/  CMMI 
/  Process 

f  M 

I  Management 


CMMI 

Engineering 


CMMI 

Project 

Management 


CMMI 

Support 


CMMI 
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Steps  for  CMMI-DEV  Practice  Implementation 
SEI Partner  f°r  Distributed  Small  Businesses 


~rl*  Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Understand  the  Integrated  Business  Objectives 


Key  growth  investments: 

CMMI-DEV,  ISO  9001-2000 
Process  training 
Collaborative  H/W  &  SA/V 
Process  engineer  support 


Collect  Lessons  Learned 

•  What  did  we  do  well? 

•  Where  were  we  lucky? 

•  What  do  we  need  to 
improve? 


Impact 

Metrics 


Market  Area 
current 
revenue/profit 
projected  growth 


Evaluate  Business  Opportunity 

“risk  of  winning” 


cost 

schedule 
technical 

I 

! 

Track  Risks 
current 
new 


/ 


Evaluate  L 

Mitigation  High,  Medium,  Low 

Plans 

Contract  Type 
new 

customer 

technical 

complexity 

staff 


Risk  Template 


CMMI 


Identify  Project  Specific  Risks 

Cost,  schedule,  technical  impacts 

-  mitigation  planning  data 

-  customer  involvement 

-  monthly  statusing 


Diligent  Consulting  Inc. 
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Strategy  Map  for  Using  Audits  &  Appraisals 
SEI Partner  f°r  Distributed  Small  Businesses 


Strategy  to  achieve  full  goal  satisfaction  in 
all  Process  Areas  for  Maturity  Level  3. 


Quality  PM  or  staff  auditing  of 
processes 


•  Internal  mini-team  analysis  of  body  of 
evidence 


•  Formal  analysis  of  body  of  evidence 

•  Assess  readiness  until  proven 


SCAMPI  C 
SCAMPI  B 
SCAMPI  A  3 


CMMI 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


Diligent  Consulting  Inc. 
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SEI  Partner 


Lessons  Learned  in  Managing  CMMI-DEV 
Practice  Implementation  for  Distributed  Small 
Businesses 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


>  Understand  that  all  process  improvement  opportunities  are 
investments  and  have  costs  and  risks  as-well-as  eventual 
returns  on  investment. 

>  Receive  sustained  management  commitment  in  providing  a 
dedicated  project  lead,  suited  support  personnel,  and 
resources. 

>  Avoid  “surprises”  by  incorporating  regular  progress  reviews 
into  weekly  senior  management  telecons. 

>  Include  distributed  site  staff  members  in  planning  and  reviewing 
practice  implementation  to  understand  the  continuity  and  costs 
and  identify  the  key  risks,  such  as  customer  culture  & 
expectations. 

>  Use  the  guidance  of  an  SEI-trained  lead  appraiser  from  the 
beginning  for  strategy  planning,  model  and  appraisal  training, 
and  appraisals. 

>  Work  from  a  thorough  Process  Improvement  Plan  which 
includes  risks,  costs,  and  schedule. 


CMMI 


Diligent  Consulting  Inc. 
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SEI  Partner 


Lessons  Learned  in  Managing  CMMI-DEV 
Practice  Implementation  for  Distributed  Small 
Business  (continued) 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


CMMI 


>  Implement  practices  with  the  use  of  consistent  and 
approved  tailored  templates. 

>  Coordinate  team  process  implementation  and  information 
sharing  across  geographically  distributed  sites  using  a 
collaborative  file  sharing  capability  e.g.,  SharePoint. 

>  Maintain  progress  and  schedule  by  conducting  weekly 
telecons  with  each  support  team  member  individually. 

>  Schedule  and  perform  regular  independent  reviews  of 
artifacts  by  the  project  lead  and  members  of  the  support 
team. 

>  Conduct  pre-readiness  reviews  before  all  appraisals. 


Diligent  Consulting  Inc. 
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SEI  Partner 


The  End 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


CMMI 


You  Have  Just  Seen: 
Challenges  and  steps  in 
implementing  CMMI-DEV 
practices  across  a 
distributed  small  business 
from  the  “30,000  feet”  level. 


Questions  or  Comments  ? 


A  III 

# 

38 

b 

DILIGENT 

Consulting  Inc. 
Diligent  Consulting  Inc . 
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Contact  Information 

SEI  Partner 


Transdyne  Corporation 
CMMI  Implementations 
in  Small  &  Medium 
Organizations 


David  L.  Walker 

Quality  Program  Manager 
david.  walker(a)_diliaent-us.  con 

DILIGENT  Consulting  Inc. 

A  Service  Disabled  Veteran  Owned  Small 
Business 

SEI  CMMI  Maturity  Level  3 

Office:  210-826-9300  Ext#  130 
Cellular:  281-455-8198 
www.  DHiaent-us.  com 


Dr.  Mary  Anne  Herndon 
Director ;  Process  Improvement 

maherndon(d).san.rr.  com 

Transdyne  Corporation 

•  Certified  Instructor  for  the  Introduction  to  the 
CMMI  vl.2 

•  SEI  Certified  Lead  Appraiser 

Voice/FAX:  (858)  271-1615 
httD://transdvnecorp.  com 


CMMI 


Diligent  Consulting  Inc - 


DILIGENT 

Consulting  Inc 
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CMMI  Bottoms -Up  Assessment:  a 
Grounded  Analysis  from  the 
Perspective  of  Practicing  Engineers  in 

Defense  Engineering 


Dr.  Bruce  Beaded 
17  November  2010 


Presentation  Topics 

Quality  Leverage  Points  Trilogy:  Process, 
People,  &  Technology 

Social  Theories  Applied  to  CMMI:  Marx,  Taylor, 
&  Weber 

CMMI  Bottoms-Up  Assessment:  a  Grounded 
Analysis  from  the  Perspective  of  Practicing 
Engineers  in  Defense  Engineering 

CMMI  Case  Study  Conclusions  & 
Recommendations 


Quality  Leverage  Points  Trilogy 
Process,  People,  Technology 


Everyone  realizes  the  importance  of 
having  a  motivated,  quality  workforce 
but... 


PEOPLE 


...even  our  finest  people 
can’t  perform  at  their  best 
when  the  process  is  not 
understood  or  operating 
“at  its  best.” 


^-Process,  People,  Technology 


Source:  SEI  CMMIV1.2 
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What  About  People? 


According  to  Watts  Humphrey: 

>  “Talented  people  are  the  most  important  element  in 
any  software  organization.” 

>“The  better  and  more  experienced  they  are,  the 
better  the  chance  of  producing  first-class 
results.... First-class  people  are  essential,  but  they 
need  the  support  of  an  orderly  process  to  do  first- 
class  work.” 

What  Can  We  Learn  about  People  from 

Social  Theories? 


Social  Theories  Applied  to  the  CMMI 

©@ 


CMMI  as  Contemporary  Iron  Cage:  a  Grounded  Analysis  from  the 
Perspective  of  Practicing  Engineers  in  Defense  Engineering.  St.  Paul, 
MN:  University  of  St.  Thomas  doctoral  dissertation.  Copyright  ©  2009 
by  Bruce  B.  Beadell.  All  Rights  Reserved 


Conflict  Theory  -  Karl  Marx 


•  Worker  Alienation  [Manuscripts  of 
1844]: 

-  “The  boss  imposes  the  kind  of  work,  the 
method  and  the  rhythm,  but  never  bothers  if 
the  worker  ends  up  as:  a  mere  appendage  of 
flesh  on  a  machine  of  iron.” 

-  “The  deepest  essence  of  man,  his  creative  act, 
has  been  transformed  into  a  possession.” 

-  “Alienation  not  only  degrades  man,  but  also 
depersonalises  him.” 
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Conflict  Theory  -  Marxian 
CMMI  Implications 


•  Companies  have  established  Mandatory  Policies, 
Procedures,  &  Work  Instructions  to  comply  with  the 
CMMI  that  Rigidly  Control  Engineering  Processes  & 
Product  Development 

-  “The  boss  imposes  the  kind  of  work,  the  method,  and  the 
rhythm  ...” 

•  Management  has  usurped  Engineering  Creativity 

-  “The  deepest  essence  of  man,  his  creative  act,  has  been 
transformed  into  a  possession.” 
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Frederick  Winslow  Taylor’s 
Scientific  Management 


Introduced  many  of  the  Alienation  Methods 
that  Karl  Marx  Warned  about: 


-  Pushed  the  Division  of  Labor  to  the  Extreme 

•  Decomposed  the  work  process  into  fragmented  assembly 
line  tasks  that  separated  the  mental  concept  of  product 
creation  from  the  physical  act  of  product  creation 

•  Claimed  the  mental  act  of  creation  for  management  & 

relegated  the  physical  act  of  production  to  the  worker 

•  Management  usurped  the  master  craftsmen’s  knowledge. 

documented  it,  codified  it  via  work  sheets  for  unskilled 
workers  to  follow,  &  eliminated  the  master  craftsmen  „ 


Frederick  Winslow  Taylor’s 
Scientific  Management 

•  Used  Scientific  Management  Principles  to 

Maximize  Management  Control  of  the  Worker’s 

Thinking,  Acting,  &  Doing: 

-  Management  seized  total  Control  of  the  Manufacturing 
Processes,  Methods  of  Product  Production,  &  Workers 
Mental  and  Physical  Movements 

-  Dehumanized,  Deskilled  and  Robotized  Workers,  which 
Alienated  them  from  their  Work  Product,  their  fellow  Workers, 
&  their  Families 

-  Used  Foremen  and  Industrial  &  Quality  Control  Engineers  as 

Work  Process  Cops  -  forcing  the  Workers  to  blindly  follow 
Scripted  Processes,  Methods,  &  Task  Sheets  9 


Frederick  Winslow  Taylor’s 
Scientific  Management  - 
CMMI  Implications 


Division  of  Labor  has  basically  occurred  in 
accordance  with  the  CMMI’s  22  Process  Areas 


Management  Control  of  the  Engineer’s 
Thinking,  Acting  and  Doing  has  been  achieved 
via  mandatory  CMMI  driven  Policies, 
Procedures,  &  Work  Instructions 

Compliance  Verified  via:  Internal  Audits, 
Process  Audits,  QA&  CM  Audits,  &  CMMI 
Appraisals 


Max  Weber’s  Conflict  Theory 


ureaucratization  of  Organizations 


-  Provided  Management  with  an  Organizational 
Methodology  that  aids  Decision  Making  & 
Maximizes  4  Factors  of  Business  Operation: 

•  Efficiency  -  achieved  using  optimal  means  (tasks, 
methods,  &  processes)  to  attain  desired  ends  (products) 

•  Calculabilitv  -  achieved  using  quantification  of  inputs, 
processes,  outputs,  resources,  &  finances 

•  Predictability  -  achieved  using  consistent,  uniform 
production  methods  &  processes 

•  Control  -  achieved  tightly  controlled  automated 
dehumanized  processes,  feedback  systems,  & 
monitoring  of  work  performance 


Max  Weber’s  Conflict  Theory  - 
CMMI  Implications 


SEI  &  DoD  have  confirmed  CMMI  Assessed 
Companies  Achievement  of  Weber’s  4  Factors 
of  Business  Operation  &  Performance: 


-  Greater  Efficiency,  Calculability,  Predictability,  & 
Control 


However,  Bureaucratization  Impacts  (Iron 
Cage)  on  Engineers  largely  Ignored: 

•  Job  Satisfaction  &  Job  Performance? 

•  Loss  of  Incentive  &  Motivation? 

•  Loss  of  Creativity? 


CMMI  Bottoms-Up  Assessment:  a 
Grounded  Analysis  from  the 
Perspective  of  Practicing  Engineers 
in  Defense  Engineering 

©@ 


CMMI  as  Contemporary  Iron  Cage:  a  Grounded  Analysis  from  the 
Perspective  of  Practicing  Engineers  in  Defense  Engineering.  St.  Paul, 
MN:  University  of  St.  Thomas  doctoral  dissertation.  Copyright  ©  2009 
by  Bruce  B.  Beadell.  All  Rights  Reserved 


Case  Study  Background 

>  Purpose:  Determine  how  CMMI  Practitioners  (Engineers) 
viewed  the  effects  of  the  CMMI  on  their  Job  Performance  & 
Job  Satisfaction  at  Defense  Engineering  (pseudonym) 

>  Defense  Engineering  went  through  2  CMMI  VI  .2  Appraisals 
during  this  case  study  period:  ML3  &  ML5 

>  Case  Study  Approach: 

❖  12  Engineers  (HW,  SW,  SE,  QA)  participated  in  study  who  had  actively 
implemented  CMMI  requirements  /  capabilities  on  their  Projects  and 
participated  in  CMMI  appraisals: 

❖  Typical  use  of  CMMI  Ranged  from  2  to  5  years 

❖  Engineering  Experience  Ranged  from  9  to  35+  years 

❖  Electronic  Questionnaire  comprised  of  10  Open-Ended  Questions, 
analyzed  &  scored  responses  (Positive  =  +5,  Neutral  =  0,  Negative  =  -5), 
and  conducted  follow-up  discussions  with  respondents: 

>  Used  Qualitative  and  Quantitative  Hybrid  Analysis  Methods 

>  Listened  to  the  Voices  of  Engineers:  emails,  discussions,  &  CMMI  training  sessions 


14 


How  CMMI  Affects  Practicing  Engineers 

Very  Favorable 


Less  Favorable 


1 


Voices  of  Engineers 

•  Q1:  How  does  CMMI  affect  an  engineering  practitioner’s  ability 
to  accomplish  engineering  work  tasks  effectively? 

✓  Results:  Positive  (P)  =  10  {83.3%},  Neutral  (N)  =  1  {8.3%},  Detrimental  (D)  = 
1  {8.3%} 

>  R2  [P]  -  “Very  positively.  There  is  no  longer  a  question  of  what  is  to 
be  done,  since  it  is  clearly  documented  in  policies,  procedures,  and 
instructions.” 

>  R6  [P]  -  “Policies,  procedures  and  instructions  significantly 
improved  the  stakeholder  commitment  from  interdependent  teams, 
resulting  in  more  effective  involvement  in  decisions  and  tasks  that 
affect  them.” 

>  R9  [D]  -  “In  all  honesty,  I  think  that  the  pursuit  of  CMMI  certification, 
has  created  such  detailed  and  strict  policies,  procedures,  and 
reporting  requirements  that  it  hinders  an  engineer’s  ability  to 
accomplish  work  tasks  effectively,  creatively,  and  efficiently  more 
than  it  has  helped.” 
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Common  Themes 


•  Q1  -  Perform  Engineering  Tasks  Effectively 

>  Positive  CMMI  Effects 

•  CMMI  specifies  what  engineering  tasks  must  be  done  via 
specific  practices  (SPs)  and  generic  practices  (GPs). 

•  Company’s  CMMI  Compliant  Policies,  Procedures,  & 
Instructions  (PPIs)  specify  how  these  engineering  tasks  are  to 
be  accomplished. 

>  Negative  CMMI  Effects 

•  CMMI  may  cause  companies  to  implement  too  many  Policies, 
Procedures,  &  Instructions  (PPIs),  which  can  overwhelm 
engineers  from  working  effectively  by  slavishly  following  these 
PPIs  unless  significant  tailoring  is  encouraged  &  allowed. 

•  Too  many  Process  Areas  (22)  and  Specific/Generic  Goals 
(116)  &  Practices  (439)  that  must  be  implemented, 
institutionalized,  and  satisfied  to  attain  CMMI  ML5. 
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Voices  of  Engineers 

•  Q2:  How  does  the  CMMI  contribute  to  an  engineer’s  ability  to 
create  quality  engineering  products? 

S  Results:  Positive  (P)  =  10  {83.3%},  Neutral  (N)  =  2  {16.7%},  Detrimental 
(D)  =  0  {0%} 

>  R2  [P]  -  “Very  positively.  Of  note  here  is  the  organizational  use  of  a 
standard  peer  review  process  that  has  done  (in  my  opinion)  the  most  to 
enhance  the  quality  of  our  Engineering  Products.  Also,  standardizing  on 
tools  has  greatly  affected  quality.” 

>  R3  [P]  -  “Increased  -  more  decisions  are  made  based  on  data  versus 
engineering  judgment,  personal  agenda,  or  management  decree.  There’s 
no  question  that  better  processes,  when  used  conscientiously,  result  in 
better,  higher  quality  products.” 

>  R5  [P]  -  “Quality  has  increased  -  Specifically  use  of  peer  reviews  in  a 
quantitative  manner  drives  up  quality.” 

>  R6  [N]  -  “Quality  of  the  processes  and  products  is  ultimately  up  to  the 
management  team.  If  the  management  team  does  not  enforce  high 
standards,  the  quality  is  shoddy  regardless  of  whether  or  not  policies, 
procedures,  and  instructions  are  being  followed.” 
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Common  Themes 


•  Q2  -  Create  Quality  Products 

>  Positive  CMMI  Effects 

•  CMMI  VER  &  VAL  -  Ensures  building  the  product  right  and 
building  the  right  product. 

•  CMMI  DAR  -  Enables  better  engineering  decisions  based 
upon  disciplined,  rigorous  quantitative  analysis. 

•  CMMI  OPD,  MA,  QPM  &  OPP  -  Require  use  of 
organizational  &  project  data,  sharing  of  measurement 
data  across  the  organization,  and  making  process/product 
improvements  based  upon  quantitative  data/information. 

>  Negative  CMMI  Effects 

•  No  Negative  Statements  Voiced  by  Engineers. 
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Voices  of  Engineers 

•  Q3:  How  does  the  CMMI  affect  the  engineer’s  ability  to 
accomplish  their  engineering  tasks  efficiently? 

^  Results:  P  =  7  {58.3%},  N  =  3  {25.0%},  D  =  2  {16.7%} 

>  R1  [P]  -  “I  think  that  the  main  way  the  CPF  [Common  Process 
Framework]  has  increased  my  task  execution  efficiency  is  with 
the  templates  they  provide....” 

>  R2  [P]  -  “Very  positively.  As  personnel  move  from  program  to 
program  their  learning  curve  is  greatly  reduced  since  tasks  are 
performed  consistently  and  the  tool  base  is  relatively  the  same.” 

>  R9  [D]  -  “We  have  created  an  environment  where  it  has  become 
more  important  and  time  consuming  to  show  that  you  have 
followed  the  process  than  it  is  to  do  the  work  and  meet  the 
customer’s  needs.” 

>  R11  [D]  -  “Our  policies,  procedures  and  instructions  do  not  scale 
to  the  size  of  our  programs.  The  process  leans  toward  large 
programs  that  can  afford  the  overhead.  Small  projects  gain  much 
less  value  from  the  implementation  of  CMMI.” 
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Common  Themes 

*  Q3  -  Execute  Engineering  Tasks  Efficiently 

>  Positive  CMMI  Effects 

•  CMMI  OPD,  DAR,  MA,  OPP,  QPM,  VAL,  &  VER  -  Ensure 
building  the  engineering  product  using  standardized 
organizational  templates:  plans,  test  procedures,  and  reports. 

•  CMMI  DAR  -  Standardized  DAR  report  form  used. 

•  CMMI  OPD,  MA,  QPM,  &  OPP  -  Require  use  of  organizational  & 
project  data,  sharing  of  measurement  data  across  the 
organization,  and  making  process/product  improvement 
decisions  based  upon  quantitative  data/information. 

>  Negative  CMMI  Effects 

•  CMMI  has  spawned  an  engineering  process  that  is  too 
burdensome  and  overly  focused  on  documenting  and 
management  reporting. 

•  Only  beneficial  to  larger  projects  that  can  afford  the  huge 
reporting/recording  overhead. 
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Voices  of  Engineers 

•  Q4:  What  degree  of  control  does  your  CMMI  compliant 
policies,  procedures  and  instructions  impose  on  your 
engineering  work  activities? 

✓  Results:  P  =  6  {50%},  N  =  2  {16.7%},  D  =  4  {33.3%} 

>  R3  [P]  -  “Higher  control  is  imposed  in  relation  to  other  non-CMMI 
organizations  where  I’ve  worked.  This  is  expected,  given  the  CMMI 
focus  on  process  repeatability/stability  and  continuous 
improvement.” 

>  RIO  [P]  -  “The  processes  are  supposed  to  control  all  aspects  of  the 
engineering  activities,  and  it  essentially  does  except  for  the  human 
variations  that  are  inevitable.” 

>  R9  [D]  -  “I  would  characterize  the  degree  of  control  that  the  policies 
impose  to  be  ‘over  the  top.’  We  have  program  reviews,  engineering 
reviews,  checkpoint  reviews,  PAVM  [Process  Asset  Verification 
Matrix]  audits,  quality  process  audits,  peer  review  reporting,  weekly 
team  meetings,  weekly  schedule  meetings,  monthly  measures, 

QMP  [Quantitative  Management  Plan]  measures,  EVMS  [Earned 
Value  Management  System]  measures,  and  the  list  goes  on.” 
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Common  Themes 


•  Q4  -  Control  Effected  Organization’s  PPIs 

> Positive  CMMI  Effects 

•  CMMI  GP  2.8  -  “Monitor  and  Control  the  Process” 

•  Control  is  necessary  for  process  repeatability,  product 
consistency,  and  predictable  schedule,  cost,  &  quality 
results. 

> Negative  CMMI  Effects 

•  Too  much  control  can  be  stifling  due  to  layers  of 
approvals  and  counterproductive  due  to  numerous  & 
burdensome  reviews,  audits,  and  meetings  that  are 
viewed  by  some  engineers  as  non-value  added. 
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Voices  of  Engineers 

•  Q5:  How  does  the  CMMI  affect  your  ability  to  think  and  act 
as  an  engineer? 

✓  Results:  P  =  4  {33.3%},  N  =  4  {33.3%},  D  =  4  {33.3%} 

>  R7  [P]  -  “  CMMI  PPI  [Policies,  Procedures,  and  Instructions] 
encourage  me  to  act  and  think  as  an  engineer.  I  am  exposed  to 
the  larger  breadth  of  the  program  with  peer  reviews,  PACAs 
Preventative  and  Correction  Action],  and  OIDs  [Organizational 
nnovation  and  Deployment].” 

>  R6  [D]  -  “Policies,  procedures  and  instructions  decrease  my 
creativity  and  my  ability  to  innovate  because  they  don’t  challenge 
me  to  think  up  my  own  solutions.  I  am  provided  a  template  for 
every  task.  I  am  provided  a  flowchart  of  process  activities  that 
must  be  followed.  I  am  treated  like  a  cog  in  a  big  wheel,  without  a 
mind  of  my  own.” 
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Common  Themes 


•  Q5  -  Ability  to  Think  &  Act  as  an  Engineer 

> Positive  CMMI  Effects 

•  CMMI  helped  shape  the  Common  Process  Framework 
used  by  engineers,  which  provides  them  with  a  defined 
structure,  instructions,  and  specific  processes  to  follow 
to  create  quality  engineering  products. 

> Negative  CMMI  Effects 

•  CMMI  adoption  reduced  and  stifled  engineers’ 
creativity,  innovation,  and  performance  due  to  its 
burdensome,  prescriptive,&  bureaucratic  processes. 
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Voices  of  Engineers 

•  Q6:  How  has  the  CMMI  affected  your  job  performance? 

■S  Results:  P  =  9  {75.0%},  N  =  2  {16.7%},  D  =  1  {8.3%} 

>  R5  [P]  -  “They  have  been  useful  when  I’ve  needed  guidance, 
yet  haven’t  been  too  restrictive  or  limiting.  Once  again,  this 
has  been  particularly  helpful  when  performing  certain 
duties/tasks  for  the  first  time.” 

>  R7  [P]  -  “PPI  [Policies,  Procedures,  and  Instructions]  aid  me 
in  consistently  creating  quality  work  products  by  peer 
pressure,  larger  program  knowledge  by  participation  in  non- 
traditional  disciplines  for  me.” 

>  R9  [D]  -  “But  if  you  ask  me  if  I  am  improving  my  overall  job 
performance  in  getting  the  tasks  done  and  getting  quality 
products  out  to  my  customer,  I’d  have  to  say  that  I  am  a  worse 
performer  than  historically  because  of  all  the  process  adders  I 
have  to  do  in  order  to  do  the  same  tasks.” 
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Common  Themes 


•  Q6  -  Engineer’s  Job  Performance 

>  Positive  CMMI  Effects 

•  CMMI  based  PPIs  contribute  to: 

-  Shorter  Learning  Curves. 

-  Performing  Engineering  Tasks  Consistently  &  Uniformly. 

-  Sharing  of  Process  &  Product  Improvements  across  Projects. 

-  Common  use  of  Process  &  Product  Measures  Information. 

> Negative  CMMI  Effects 

•  CMMI  based  PPIs  are  perceived  as  detrimental  due  to 
the  numerous  non-value  added  tasks  imposed. 

•  Large  number  of  PPIs  that  are  invoked  on  the  Project. 
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Voices  of  Engineers 

•  Q7:  How  has  the  CMMI  affected  your  job  satisfaction? 

■f  Results:  P  =  6  {50.0%},  N  =  3  {25.0%},  D  =  3  {25.0%} 

>  R2  [P]  -  “It  has  made  my  job  more  satisfying.  I  tend  to  take  a 
structured  approach  to  tasks  and  problems  which  is  right  in  line 
with  the  CMMI.” 

>  R3  [P]  -  “It  has  improved  my  job  satisfaction  significantly.  ...I 
prefer  working  in  an  environment  with  clear  methods  based  on 
best  practices  and  adequate  data.” 

>  R9  [D]  -  “I  go  home  frustrated  at  the  barriers  to  efficient 
performance  and  the  total  lack  of  empowerment  that  I  have  to 
make  it  happen.  I  have  created  more  documents  and  minutes 
and  measures  in  the  past  several  years  that  no  one  looks  at 
once  they  are  created.” 
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Common  Themes 


•  Q7  -  Level  of  Job  Satisfaction 

> Positive  CMMI  Effects 

•  Organization’s  CMMI  based  PPIs  Provide: 

-  Structured  Approach  and  Clear  Methods  for  Engineering  Tasks. 

-  Based  upon  Best  Industry  Engineering  Practices. 

-  Ensures  consistent  Project  Execution  across  the  Organization. 

> Negative  CMMI  Effects 

•  Organization’s  CMMI  based  PPIs  have: 

-  Diminished  Efficiency  &  Empowerment  due  to  Required  Non- 
Value  added  Tasks. 

-  Hampered  Getting  High  Priority  Work  Accomplished. 

-  Imposed  a  Significant  Overhead  Burden  on  Projects. 
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Voices  of  Engineers 

•  Q8:  How  has  the  CMMI  affected  your  decision  making? 

■s  Results:  P  =  9  {75.0%},  N  =  3  {25.0%},  D  =  0  {0.0%} 

>  R1  [P]  -  “DARs  [Decision  Analysis  and  Resolution]  and  PACAs 
[Preventive  and  Correction  Action]  are  effective  tools  that  allow 
you  to  at  least  partially  quantify  technical  decisions.  I  have 
been  involved  in  both  DARs  and  PACAs  and  think  they  are 
effective.” 

>  R3  [P]  -  “Increased  -  more  decisions  are  made  based  on  data 
versus  engineering  judgment,  personal  agenda,  or 
management  decree.  There’s  no  question  that  better 
processes  and  better  (measurement)  data,  when  used 
appropriately,  result  in  better,  more  informed  and  more  valid 
product  decisions.” 

>  R6  [P]  -  “Policies,  procedures  and  instructions  helped  to  lay 
the  foundations  for  decision  analysis  and  resolution  that  is 
based  on  weighted  criteria.  Overall  the  DAR  process  results  in 
the  fairest  outcomes.” 


Common  Themes 


•  Q8  -  Decision  Making  Ability 

> Positive  CMMI  Effects 

•  Implemented  CMMI  DAR  &  CAR  Processes: 

-  Powerful  tools  that  Facilitate  Good  Decision  Making  due  to 
»  Structured  Process  to  follow. 

»  Fairest  of  Outcomes  based  on  Quantitative  Criteria. 

»  Robust  Root  Cause  Analyses  &  Defect  Prevention. 

> Negative  CMMI  Effects 

•  No  Negative  Statements  Voiced  by  Engineers. 
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Voices  of  Engineers 

•  Q9:  How  has  the  CMMI  affected  your  empowerment? 

>  Results:  P  =  8  {66.7.0%},  N  =  1  {8.3%},  D  =  3  {25.0%} 

>  R1  [P]  -  “The  ability  to  tailor  is  very  empowering,  if  you  are 
willing  to  do  it.” 

>  R4  [P]  -  “The  CMMI  demands  collection  of  process  artifacts 
and  metrics.  Development  of  these  items  helps  structure  the 
thought  process  and  provides  a  broader  view  of  the  product.” 

>  RIO  [D]  -  “I  do  not  believe  that  empowerment  is  granted  by  the 
process.  It  seems  that  the  process  outlines  how  engineering 
decisions  are  made  and  who  has  to  approve  them.  Everything 
has  to  be  checked/approved  by  someone  else,  so  consequently 
I  don’t  feel  empowered  at  all.” 
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Common  Themes 


•  Q9  -  Feelings  of  Empowerment 

> Positive  CMMI  Effects 

•  CMMI  Based  PPIs  Provide: 

-  Ability  to  Tailor  Processes  &  Practices. 

-  Use  of  Integrated  Product  Teams  (IPTs). 

-  Collection  of  Common  Artifacts,  Measures,  and  Data. 

-  Team  Involvement  in  DAR  (Decision  Analysis  &  Resolution) 
Project  Decisions. 

> Negative  CMMI  Effects 

•  CMMI  Based  PPIs  Provide: 

-  Fear  of  Process  &  Product  Audit  Noncompliance  Findings. 

-  Inhibits  Empowerment  due  to  Multiple  Levels  of:  Checking, 
Verifications,  and  Approvals. 
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Voices  of  Engineers 

•  Q10:  How  has  the  CMMI  affected  your  Creativeness? 

■S  Results:  P  =  7  {58.4%},  N  =  1  {8.3%},  D  =  4  {33.3%} 

>  R1  [P]  -  “A  frequent  criticism  that  I  hear  is  that  they  ‘limit 
creativity’ ...  Ido  NOT  think  this  is  the  case,  especially  with  well 
thought  tailoring.” 

>  R7  [P]  -  “Creativity  is  enhance[d]  by  exposure  to  larger  scope  of 
program.” 

>  R6  [D]  -  “Policies,  procedures  and  instructions  restrict  my 
creativity  by  not  allowing  me  to  try  different  solutions  that  I  feel 
would  work  better.” 

>  R9  [D]  -  “It  has  stifled  it  to  a  great  extent.  We  spend  more  and 
more  time  measuring,  documenting,  reporting  and  convincing 
than  we  do  actually  doing  the  real  engineering  work.” 
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Common  Themes 


•  Q10  -  Feeling  of  Creativeness 

>  Positive  CMMI  Effects 

•  Tailoring  of  PPI  Processes  Enables  Creativity. 

•  Consistent  use  of  PPI  Processes  allows  Creativity  in 
Engineering  Solutions. 

•  Common  Process  Framework  (CPF)  Flexibility  Encourages 
Creativity. 

>  Negative  CMMI  Effects 

•  CPF  Restricts  use  of  Non-standard,  Different,  or  Innovative 
Engineering  Solutions. 

•  Excessive  Burden  of  Measuring,  Documenting,  &  Reporting 
Stifles  Creativity. 

•  Creativity  does  not  Spring  from  Process  but  is  Intrinsic  to  an 
Engineer. 


35 


CMMI  Case  Study 
Conclusions  & 
Recommendations 


36 


People  Implications 

•  What  Really  Motivates  People? 

•  According  to  Daniel  H.  Pink  “Drive”  there  are  3 
Fundamental  Intrinsic  Motivational  Elements: 

>  Purpose  -  “Live  a  Life  of  Purpose” 

❖  Have  worthwhile  goals  to  strive  for  and  achieve! 

>  Autonomy  -  “Direct  Our  Own  Lives” 

❖  Become  empowered  to  think,  act,  and  achieve  your  goals! 

>  Mastery  -  “Extend  and  Expand  Our  Abilities” 

❖Have  the  determination,  capacity,  and  ability  to  build: 
capable  processes,  reliable  products,  and  valuable 
relationships! 
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How  CMMI  Affects  Practicing  Engineers 

Very  Favorable 


Applying  Pink’s  Intrinsic 
Motivational  Elements  for 
People:  P  =  Purpose, 

A  =  Autonomy,  M  =  Mastery 


Less  Favorable 
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Conclusions 

•  CMMI  Suboptimization:  CMMI  is  overly  Focused 
on  1  Element  (Process)  of  Engineering  Product 
Development  Framework: 

> People  Execute  Processes  to  Build  Products! 

> People  Make  Processes  Viable  &  Successful! 

>  People  are  the  Prime  Movers  &  Users  of 
Processes  &  Technologies! 

> People  ->  Processes  ->  Technology:  Are  the 
CMMI  Silver  Bullets  that  Fred  Brooks  denied,  but 
only  when  synergistically  blended  within  the 
organization’s  Engineering  Product  Development 
Framework! 
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Recommendations 

•  How  can  we  help  Engineers  (CMMI  Practitioners) 
better  Utilize  &  Practice  the  CMMI  to  Improve  their 
Job  Performance  &  Job  Satisfaction? 

>  Watts  Humphrey  TSP  (Team  Software  Process)  CMMI 
Practitioners 

>  Agile  Processes  ->  CMMI  Practitioners 

>  Six  Sigma  Toolkit  ->  CMMI  ML4  &  ML5  Practitioners 

>  People  CMM  ->  CMMI  Practitioners 

>  Daniel  Pink  (“Drive”)  understand  what  Motivates  People: 
Purpose,  Autonomy,  &  Mastery  ->  CMMI  Practitioners 

>Dr.  B  ->  Provide  Acknowledgement,  Recognition,  & 
Appreciation  ->  CMMI  Practitioners 

>  Think:  People  (1st)  ->  Processes  (2nd)  ->  Technology  (3rd) 


Questions  ? 
Comments? 
Thank  You! 


Backup  Slides  &  Supporting  Information 
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Case  Study  Approach  &  Questions 

Selected  10  CMMI  attributes  &  assessed  how  engineers  perceived  how  these 
attributes  affected  their  job  performance  and  job  satisfaction: 

1.  How  has  the  use  of  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your 

ability  to  accomplish  engineering  work  tasks  effectively? 

2.  How  has  the  use  of  your  CMMI  compliant  policies,  procedures  and  instructions  affected  the 

quality  of  the  engineering  products  that  you  create? 

3.  How  has  the  use  of  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your 

ability  to  accomplish  engineering  work  tasks  efficiently? 

4.  How  would  you  characterize  the  degree  of  control  that  your  CMMI  compliant  policies, 

procedures  and  instructions  impose  on  your  engineering  work  activities? 

5.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your  ability  to 

think  and  act  as  an  engineer? 

6.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your  job 

performance? 

7.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your  job 

satisfaction? 

8.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your  ability  to 

make  valid  engineering  product  decisions? 

9.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your 

empowerment  to  make  good  engineering  decisions? 


10.  How  have  your  CMMI  compliant  policies,  procedures  and  instructions  affected  your  creative 
ability  in  doing  your  engineering  work  tasks?  44 


Case  Study  Approach  Scoring 

Engineers’  attitudes  or  feelings  about  a  CMMI  factor 
were  grouped  into  3  response  categories:  Positive 
(P),  Neutral  (N),  &  Detrimental  (D).  These  categories 
have  the  following  meanings  &  values: 

-  “P”  means  that  this  CMMI  attribute  has  a  strong  perceived 
positive,  constructive,  and  desirable  effect  on  the 
respondent’s  work  activities  (cognitive  and  /  or  behavior). 
Assigned  value  =  +5. 

-  “N”  means  that  this  CMMI  attribute  has  virtually  no 
perceived  effect  (i.e.  Neutral)  on  the  respondent’s  work 
activities  (cognitive  and  /  or  behavior).  Assigned  value  =  0. 

-  “D”  means  that  this  CMMI  attribute  has  a  strong  perceived 

detrimental,  negative,  or  undesirable  effect  on  the 
respondent’s  work  activities  (cognitive  and  /  or  behavior). 
Assigned  value  =  -5.  45 


Table  2  -  Rank  Ordering  of  CMMI  Attributes  Matrix 
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CMMI  (VI  .2)  -  5  Maturity  Levels 

Totals:116  Goals  (SG  +  GG)  &  439  Practices  (SP  +  GP) 


Maturity  Level  Focus 

22  Process  Areas 

5  Optimizing 

Continuous 
Process 
Improvement  & 
Defect 

Prevention  [2] 

(OID)  Organizational  Innovation  and  Deployment 
(CAR)  Causal  Analysis  and  Resolution 

4  Quantitatively 
Managed 

Quantitative 
Management  [2] 

(OPP)  Organizational  Process  Performance 
(QPM)  Quantitative  Project  Management 

3  Defined 

Engineering 
&  Organization 
Process 

Standardization  - 
Qualitative 
Management  [11] 

(RD)  Requirements  Development 
(TS)  Technical  Solution 

(PI)  Product  Integration 
(VER)  Verification 
(VAL)  Validation 

(OPF)  Organizational  Process  Focus 

(OPD)  Organizational  Process  Definition  +  IPPD 

(OT)  Organizational  Training 

(IPM)  Integrated  Project  Management  +  IPPD 

(RM)  Risk  Management 

(DAR)  Decision  Analysis  and  Resolution 

2  Managed 

Basic 

Project 

Management  [7] 

(RM)  Requirements  Management 

(PP)  Project  Planning 

(PMC)  Project  Monitoring  and  Control 

(SAM)  Supplier  Agreement  Management 

(MA)  Measurement  and  Analysis 

(PPQA)  Process  and  Product  Quality  Assurance 

(CM)  Configuration  Management 

1  Initial 

Ad  Hoc  & 
Chaotic  [~0] 

Few  Processes  Documented  or  Followed  -  Land  of 
the  “Cowboys  &  Cowgirls”  -  “Don’t  Need  No 

Stinking  Processes!” 
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Assured  Communications^ 
Anytime.  Anywhere. 


Pro  cess  I  m pro  vem  en  t 
by  the  Numbers 


Debra  Perry  and  J  im  Goss 
Harris  Corporation 
November  17,  2010 


Providing  Value  To  Our  Customers 


Intelligence,  surveillance,  and  Space  and  ground  satellite  Operations  and  support  services 

reconnaissance  communications  systems 


Mission  Critical  Solutions.  Anytime.  Anywhere. 
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Agenda 


•  Background 

•  Lean  Six  Sigma 

•  Situation 

•  Process 

•  Results 

•  Conclusion 

•  Suggestion 
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Background 


•  Process  Improvement  is  a  learned  skill 

•  To  increase  awareness  and  capabilities  Harris  adopted 
Lean  Six  Sigma  (LSS) 

-  Encourages  team  members  to  look  for  ways  to  improve 
processes  by 

•  Quantifying  the  process 

•  Recommending  a  change 

•  Measuring  the  improvement 

-  Teaches  skills  to  assist  in  the  efforts 

•  Lean  Fundamentals  -  eliminate  waste 

•  Simulation  -  understand  system  performance 

•  Change  behavior  -  people  skills 

•  Six  Sigma  tools  -  statistical  skills  to  reduce  variability 
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Lean  Six  Sigma  is 


A  set  of  principles,  concepts,  and  techniques  designed  to 
enable  key  processes  to  produce  an  optimum  system  that 
will  deliver  to  our  customers: 

Exactly  what  they  need 

When  they  need  it 

In  the  quantity  they  need 

In  the  right  sequence 

Without  defects 

And  at  the  lowest  possible  cost 
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Lean  Six  Sigma  DMAIC  Process 
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Challenge 


•  To  become  LSS  Green  Belt  certified  requires  two 
process  improvement  projects 

•  What  steps  were  used  to  identify  the  projects: 

-  Investigate  areas  where  processes  are  repeated  and  used 
by  more  than  one  person 

-  Apply  Lean  Six  Sigma  principles  to  areas  where  you 
currently  work 

-  Investigate  ways  to  reduce  waste  in  processes 

-  Investigate  ways  to  take  less  time  to  accomplish  a  task 
without  losing  quality 

•  Division  Process  Group  (DPG)  is  responsible  for 
maintaining  the  Division  Program  Review  Template 
(PRT),  investigate  ways  to  improve  that  process 
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Situation 


•  Policy  requires  programs  to  prepare  an  extensive 
package  for  Program  Reviews  (PR)  each  month 

•  Problem  Statement 

•  PR  preparation  is  4  -  8  hours  per  program  per  month  per 
Program  Finance  Analyst  (PFA):  ~7200  hrs/yr 

(~100  program  *  12  months  *  6  hours  per  month) 

•  Goal 

•  Reduce  PFA  PR  preparation  effort  by  30% 

•  Measurement 

•  PFA  PR  preparation  effort 

•  Benefits 

»  Reduced  program  cost 
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PRT  Constraints 


•  A  Business  Intelligence  Solution  would  be  more 
elegant 

-  Collect  all  data  in  a  Data  Warehouse 

-  Use  Business  Intelligence  Tools  to  automatically 
produce  dashboards  and  charts,  perform  trend 
analysis  across  programs  at  different  levels  of  the 
company 

•  Can’t  invest  in  a  new  tool  at  this  time 

•  Data  must  be  gathered  from  many  different 
systems  and  entered  manually  or  cut  and  paste 

•  Can  we  improve  the  process? 


Process  Improvement  by  the  Numbers 


assuredcommunications 


NDIA  CMMI®  Conference  -  9 
15-18  November  2010 


Project  Context 


Summary  evaluation  & 
explanations, 
accomplishments, 
milestones  and  deliveries 
Issues  /  Actions 
Objectives  /  status 
Opportunities 


MPM™  data 
Headcount 
Contract/Budget  data 
MPAR  data 
Investment 
Cost  (Period  &  ITD) 
Award  Fee  Data 
PEM  data 
Supplier  data 
Etc. 


Program  Financial  Analyst 
(PFA) 


Program  Manager  (PM) 

Schedules 
Key  events/meetings 
Action  items 
Other  program  specific 
items 
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PRT  Example 


Prog  ram  Marne:  Rett  o  us 
Contract  Type:  CFAF 
Contract  -Start:  May-2333 


Customer  NO  A  CMS  PA# 

PM:  John  Doe  TCPI 

Co  ntract  E  nd:  May-20 1 9  Rejj  la  n  \  H]  Latest  Date 


160G  Co  ntract  Value:  S701.955.5K 

I.ODg]  Available  Funds:  346.317.9K  [§] 

|1|  uaajaflm  Repo rtirtg  Period:  Jj-10 


Prog  ram  Desc  r  iptio  n: 

Design ,  Deve  Sop  rent,  a  nd  Operate  the  G  'o  und  Segne  nt.  Co  nssts  of  E  nte  'o-se  Ma  nage^ne  nt  (HOC),  Miss  o  n  Ma  nage  nnent  [Sate  te  Co  mmand  &  Co  ntoJ),  P  'od  uct 
Gene'aton  [Data  Process  ng},  a -pd  F rcd  jet D  stnb uto n  [Data  D stribufor). 


Key  Area 


I'V  AaatiSIVtNI 


•:C|  indicatfscr 


Co  m  merits/ Exp  fanatic  n 


Customer  Relations 


□ 


□ 


Cost  Performance 


Lil** 


in  « 


Good.  Continuefrankand opendiscussions. 

PD  CPI:  1.14  |J  PD  CPI  w.'t)  Mtri:  Q  IT D  CPI:  1,33  Q 

Underrunforperiod  mainly  dueto  unplannedsummerva  cations 


ITDCPIwi'oWUrl: 


PD  iPI:  1.M  [oj 

On  Track 


ITD  SPI:  l.flfl  [oJ  ITD  5  PI  w,'o  Mtri : 


PD  5  PI  nto  Mtrl: 


Schedule  Performance 


■< 


Technical  Performance 


E 


Element  Specs  delivered  on-time.  SWRRs  began  on  schedule. 


Staffing 


Staffing  is  on-track.  Future  concern  is  staff  ing  for  additional  studies,  and  impact  of  Antenna  award. 


Previous:  I375.4K  HI  Current:  It. 497. 7  K  "E 

RW3/ECP04  planning  complete.  Bottoms-up  EAC  delayed  until  nearer  PDRto  accommodate  AER. 


PMB  Variance  at 
Completion 


Quality 


LastQA  PCM  Audit  Aug-JAM 

Continue  to  track  CARDS  (closures  slightly  ahead  of  schedule}.  PCM  in  green.  Yellow  continues 
due  to  number  of  open  defect  action  items  vs.  total  action  items  (ratio  in  formula;. 


Risk  Exposure 


iji» 


l_0:a« 


Material 


■< 


Wo  issues. 


Drawings 
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Initial  Assessment 


•  Initial  Projected  Statistics 

•  Initial  DPMO:  5000 

•  Current:  6  hours  *  60  min  =  360  min 

•  Desired:  3  hours  *  60  min  =  180  min 
360-  180  =  180 

•  (180  min  *  1,000,000)/(100  programs*  360  min)  =  5000 

•  Estimated  COPQ:  $126K 

•  3  hours  *  $35/hour  =  $105 

•  $105  *  12  months  *  100  programs  =  $126K 

•  Target  Projected  Statistics 

•  DPMO  Reduction  Goal:  30  % 

•  Target  DPMO:  2000 

•  30%  *  360  min  =  108  min 
180-108  =  72 

(72  *1,000,000)/(100  programs*  360  min)  =  2000 

•  Target  Savings/Revenue:  would  be  $76K/year 

1 08  min/60min/hr  *  $35/hour  *  1 2  months  *  1 00  programs  =  $76K 
but  due  to  investment  cost  only  $1 2K  per  year  -  See  Slide  1 7 

•  Benefit/Assumptions 

•  More  efficient  Program  Review  preparation  process 
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Desired  Improvement 


Program  Review  Preparation  Process 
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Brainstorming 


Define 


_ IV. _ K.  _ _ l\, _ K 

Measure  Analyze  ^Improve  Conirol  ;> 


•  Facilitated  two  PFA  Round  Tables  sessions  with  team  resulting  in  33 
improvement  suggestions 

o  Identified  2  possible  Lean  applications 

■  Reduce  effort  to  gather  data  and  populate  PRT 

■  Reduce  number  of  worksheets  being  populated 

o  No  measurement  breakdown  available  for  PRT  preparation  steps 

1 .  Collection  of  data 

2.  Population  of  PRT 

3.  Analyze  and  Review  PRT,  (correct  if  needed) 

4.  Create  PPT 

5.  Review  PPT,  correct  if  needed) 

■  Need  to  ‘instrument’  the  process 


Effort  data  is  currently  only  verbal  estimates 
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Improvement  Basis 


Define 


_ IV. _ K.  _ _ l\, _ K 

Measure  Analyze  ^Improve  Control  ;> 


PRT  Improvement  Preparation  Effort 

•  Create  PFA  Survey  to  establish  baseline  preparation  times 

•  Round  table  indicated  4-8  hours  per  month  preparation 

•  Approximately  100  programs  use  PRT  each  month 

•  Improvement  savings  estimate  of  30  -  60  minutes  each  month  per 
program 

•  Conservative  improvement  savings  estimate  (30  minutes): 

•  0.5  hours  *  12  months  *  100  program  *  $35/hr  =  $21K/year 

•  Expected  improvement  savings  estimate  (60  minutes): 

•  1.0  hours  *  12  months  *  100  program  *  $35/hr  =  $42K/year 

•  Aggressive  improvement  savings  estimate  (108  minutes) : 

•  1.8  hours  *  12  months  *  100  program  *  $35/hr  =  $75.6K/year 


Survey  to  establish  baseline  and  measureable  improvement 
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Financial  Analysis 


Projected  Savings 

Projected  Cost 

Labor 

$42K/year 

Labor 

~$150K 

Equipment 

$0K 

Equipment 

$0K 

Material 

$0K 

Material 

$0K 

Other 

$0K 

Other 

$0K 

Net  Projected  Savings  amortized  over  5  years 

$42K/year  *  5  years  =  $21  OK 
$21  OK  -  $1 50K  =  $60K  net  savings 
$60K/5  years  =$12K  annual  savings 

for  first  5  years 
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Measurement  £ 


•  Drafted  Initial  PFA  Survey 

•  Anonymous  Survey  Distributed 

-  200  PFAs  received  survey 

-  76  responded 

-  51  use  the  PRT 


Good  response  rate,  shows  high  interest 
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Measurement  Data  Analyzed 


Define 


Measure 


Analyze 


I mprove  Control  > 


•  Data  indicated: 


Time  Spent  Collecting  Data 

AVG.  192  min. 

STD.  DEV.  203  min. 

Number  PeopleSoft  Reports  Generated 

AVG.  6.5 

STD.  DEV.  2.2 

Time  Spent  Generating  PPT  Slides 

AVG.  41  min. 

STD.  DEV.  30  min. 

Not  all  programs  are  using  PRET 

29/47  =  61% 


Data  showed  room  for  improvement 
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Planned  Improvement  Effort 


Define  Measure^/  Analyze^/ 

- K 

Improve  > 

Control 

•  Process  Improvement  Project  submitted  and  funded 

•  Create  Working  Group  with  PFAs,  PeopleSoft  Engineers 
and  Improvement  Engineers  to: 

-Create  automatic  PeopleSoft  Import  function  for  PRT 

-Update  documentation  (Program  Review  Instructions 
Manual) 

-Create  roll  out  training 

•  Other  improvements: 

-  Update  PRET 

-  Delete  unused  worksheets  in  PRT 

-  Improve  PRT  analysis  capability,  remove  indirection 

Pilot  Programs  Liked  Improvement 
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Worked  Improvement  Effort 


Define  Measure^/  Analyze^/ 

- K 

Improve  > 

Control 

*  Updated  PRT,  PRET  and  documentation 

*  Created  training,  PFAs  to  present  training 

*  Piloted  new  process  with  6  programs 

*  Improvement  demonstrated 

*  Rolled  out  process  to  division  (166  trained) 

*  New  process  in  use  for  several  months 

*  Anonymous  Follow  up  survey  distributed 

-  200  PFAs  received  survey 

-  23  responded 

-  19  use  the  PRT 

Response  rate  much  lower 
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Control 


K 

_ K 

_ K  _ K 

Define  > 

Measure  >  Analyze  >  Improve  > 

1/ 

i/  k  1/ 

Follow-up  Survey  data  analyzed 

Data  indicated: 

NEW 

(OLD) 

Time  Spent  Collecting  Data 

AVG. 

100  min. 

(192  min.) 

STD.  DEV. 

79  min. 

(203  min.) 

Number  PeopleSoft  Reports  Generated 

AVG. 

4.2 

(6.5) 

STD.  DEV. 

3.2 

(2.2) 

Time  Spent  Generating  PPT  Slides 

AVG. 

68  min. 

(41  min.) 

STD.  DEV. 

65  min. 

(30  min.) 

Not  all  programs  are  using  PRET 

14/18  =  78% 

(29/47  =61%) 

PFA  Data  Collection  Time  reduced  on  average  48%!! 
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Financial  Analysis 


Actual  Savings 

Actual  Cost 

Labor 

$63K/year 

Labor 

$144K 

Equipment 

$0K 

Equipment 

$0K 

Material 

$0K 

Material 

$0K 

Other 

$0K 

Other 

$0K 

Net  Projected  Savings  amortized  over  5  years 

$63K/year  *  5  years  =  $31 5K 
$315K  -  $144K  =  $171K  net  savings 
$171 K/5  years  =$34K  annual  savings 

for  first  5  years 
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Control  -  cont.  - 


Define  Measure  Analyze  improve  Control 


•  Educate  Executive  Management,  PMs  and  PFAs  on 
improved  process 

-42  PMs  Trained 
-  1 19  PFAs  Trained 
-5  Others  Trained 

*  Modify  expectations  for  Program  Reviews 


Improvement  Sustained 
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Results 


•  Lean  Six  Sigma  project:  SUCCESSFUL! 

-  Measurable  improvement  was  achieved  for  preparation 
times  for  Program  Review  Package 

•  Improved  quality  of  process  a  bonus 

-  Some  programs  were  not  following  the  standard  process  in 
creating  their  financial  data 

-  Automating  the  process  flushed  out  some  of  the 
inconsistencies  and  helped  improve  understanding  and 
use  of  the  standard  process 

-  Improved  documentation  increased  understanding 

-  Training  in  the  new  process  pointed  out  non-standard 
pitfalls 

•  Additional  improvements  were  also  beneficial 
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Conclusions 


•  Lean  Six  Sigma  training: 

-  Helps  people  look  at  things  differently  and  question  habits 

-  Helps  people  look  for  ways  to  improve  how  we  do 
business 

-  Provides  people  with  tools  to  enable  facilitating  change 

•  People  skills 

•  Mathematical  skills 

•  Modeling  skills 

•  Increased  awareness  of  available  resources 

•  Supply  Chain  Operations  (SCO)  Center  for  Excellence 

-  Provides  Lean  Six  Sigma  training 

-  Has  data  to  show  it  pays  for  itself! 
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Suggestion 


•  People  do  not  naturally  look  for  process  improvement 

•  People  need  to  be  trained  to  think  critically 

•  If  your  company  is  not  familiar  with  Lean  Six  Sigma 
processes,  you  should  consider  adding  them 
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Contact  Information 


Harris  Corporation 
P.0.  Box  37 

Melbourne,  Florida  32902-0037 


http://www.harris.com/ 

SEI  Partner 


Debra  Perry 
dperrv@harris.com 

321-727-6830 

Jim  Goss 
iqoss@harris.com 

321-727-6830 
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Backup  Slides 
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Six  Sigma  DMAIC  Process 


•  Define  the  problem,  the  voice  of  the  customer,  and  the  project  goals, 
specifically 

•  Measure  key  aspects  of  the  current  process  or  product  and  collect 
relevant  data 

•  Analyze  the  data  to  investigate  and  verify  cause-and-effect 
relationships.  Seek  out  root  causes  of  the  defect  under  investigation 

•  Improve  or  optimize  the  current  process  or  product  based  upon 
data  analysis  using  techniques  such  as  design  of  experiments,  poka 
yoke  or  mistake  proofing,  and  standard  work  to  create  a  new,  future 
state  process  or  product.  Set  up  pilot  runs  to  establish  process 
capability 

•  Control  the  future  state  process  to  ensure  that  any  deviations  from 
target  are  corrected  before  they  result  in  defects.  Implement  control 
systems  such  as  statistical  process  control,  production  boards,  and 
visual  workplaces,  and  continuously  monitor  the  process 
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Definitions 


•  PM 

-  Program  Manager 

-  Responsible  for  execution  and  reporting  for  program 

•  PR 

-  Program  Reviews 

-  Review  of  program  status  with  upper  management 

•  PFA 

-  Program  Financial  Analyst 

-  Provides  financial  status  for  program  review  package 

•  PRT 

-  Program  Review  Template 

-  Macro  enabled  Excel  spreadsheet  that  collects  PR  data 

•  PRET 

-  Program  Review  Extraction  Template 

-  Macro  enabled  PowerPoint  file  that  creates  slides  from  PRT 

•  LSS 

-  Lean  Six  Sigma 
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More  Definitions 


•  DPMO 

-  Defects  per  Million  Opportunities 

-  A  measure  of  process  performance 

•  COPQ 

-  Cost  of  Poor  Quality 

-  Costs  that  would  disappear  if  systems,  processes,  and  products  were  perfect 

•  PEM 

-  Program  Engineering  Metrics  tool 

-  Web-based  tool  that  collects  and  reports  engineering  metrics 

•  MPM™ 

-  Deltek  MPM™ 

-  Tool  for  program-based  earned  value  management  and  reporting 

•  ITD 

-  Inception  to  Date 

•  PeopleSoft 

-  Oracle  financial  tool  to  track  labor,  costs,  etc. 
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Turning  a  Super  Tanker: 
Process  Improvement 

Change  at  NSA 


Steve  Tobin,  NSA  Way  Program  Manager,  DRC 
Michael  Mangieri,  NSA  Way  Project  Coach,  BTI 
Sue  LaFortune,  Government  Lead,  NSA 


Who  is  NS  A? 


►  The  National  Security  Agency  was  established  by 
the  Secretary  of  Defense  on  November  4,  1 952. 

►  NSA  is  part  of  the  US  Department  of  Defense 

°  The  Director  of  NSA  (DIRNSA)  is  always  a  general  officer 
of  grade  0-9  (occasionally  higher)  from  any  military 
service. 

°  The  Deputy  Director  of  NSA  (D-DIR)  is  always  a  DoD 
civilian  employee. 

►  NSA’s  core  missions  are  to  protect  U.S.  national 
security  systems  and  to  produce  foreign  signals 
intelligence  information: 

°  Information  Assurance 
°  Signals  Intelligence 
°  Network  Warfare 


Three  Key  Missions 


The  Information  Assurance  mission 
confronts  the  formidable  challenge  of 
preventing  foreign  adversaries  from 
gaining  access  to  sensitive  or  classified 
national  security  information. 

The  Signals  Intelligence  mission  collects, 
processes,  and  disseminates  intelligence 
information  from  foreign  signals  for 
intelligence  and  counterintelligence 
purposes  and  to  support  military 
operations. 

NSA  enables  Network  Warfare  operations 
to  defeat  terrorists  and  their 
organizations  at  home  and  abroad. 


B 


NSA’s  Mission  and  Systems 
Development 


►  The  Information  Assurance,  Signals 
Intelligence,  and  Network  Warfare  missions 
are  highly  technical. 

°  Systems  development  and  integration— particularly 
software  intensive  systems— are  a  key  enabler  in 
fulfilling  these  missions. 

►  NSA  has  many  software  development 
projects,  ranging  from  both  the  very  large 
(1 00s  of  developers)  to  the  very  small 

(1  -3  developers). 


NSA  Process  Improvement 
History 

►  Early  1  990’s  -  early  2000’s: 

°  JACKPOT  and  Software  Engineering  Knowledge  Base 
Center 

►  2003  -  2008: 

°  LIONSHARE 

•  One  (small)  organization  achieved  CMMI®  ML  2  and 
recently  CL  3  in  selected  Process  Areas 

°  Six  Sigma  (and  Lean  Six  Sigma) 

►  2008  -  Present: 

NSA  Way 


o 


is  the  “NSA  Way”? 

►  “The  NSA  Way  is  a  unified  framework  for 
building  large,  complex,  primarily  software 
systems  that  meet  the  diverse  needs  of  NSA 
missions.  It  is  lightweight,  intuitive,  and 
independent  of  project  size  and  development 
methodology.” 

►  It  is: 

°  Based  on  a  Customer/Supplier  theme 
°  Focused  on  outputs  over  processes 
°  About  continuous  improvement 
°  Applicable  in  Agile,  Iterative,  and  Waterfall  LCMs 
°  Independent  of  team  size 


How  does  NSA  Way  Fit  in  NS  A? 

►  NSA  Way  defines  core  expectations  for  how 
systems  and  software  development  are  done. 

►  NSA  Way  deploys  ‘coaches’  into  NSA  systems  and 
software  development  organizations  to  provide 
implementation  guidance  and  to  assess  progress 

►  NSA  Way  is  implemented  through: 

Gates  (Life  cycle  control-milestones) 

°  Processes  (currently  there  are  7) 

Metrics  (Quality  related) 

►  Driving  cultural  and  behavioral  change  first, 
process  maturity  second 


Contrast  Between  NSA  Way  and 
past  NSA  PI  Initiatives 


Senior  Management 
Support 

Participation  by 
respected  technical 
leaders 

Based  on  CMMI  (or 
other  model) 


Appraisals  / 
Assessments 


Examples  from  Past  Initiatives 

NSA  Way 

Supported  by  a  'junior'  senior  leader  who 
could  not  influence  other  key  stakeholders. 
No  clear  champion  other  than  the  sponsor 

•  Supported  at  the  highest  level 

•  Championed  by  senior 
personnel  both  within  key 
offices  and  across  the 
organization. 

Limited  or  none 

Developed  and  supported  by  some 
of  the  organization's  technical  leads. 

Explicitly 

Implicitly 

Participating  programs  went  through  an 
initial  SCAMPI  class  A  appraisal  as  an 
entry  condition  for  participating. 

Monthly  assessments  directly 
connected  to  limited  areas  where 
technical  leads  agree  there  is 
positive  return-on-investment. 

Gates 


External  Application 
Development  and 
Testing 


•  Suppliers  and  customers  at  each  gate 

•  Gate  Criteria  represents  what  the 
customer  needs  to  be  successful 


Planning 


Customer 
Signoff  / 
Deployment 


Product  Development, 
Integration  and  Testing 


Unit  Testing 


Lessons  Learned 


Code  Reviews 


Coding  Standards 


Requirements  Management 


Configuration  Management 


Defect  Management 
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NSA  Way  Required  Activities 

►  ‘Establish  and  maintain 

°  Coding  Standards 

°  Configuration  Management  Processes 
°  Internal  and  External  Interface  Specifications 
°  Verification  processes 

•  Conduct  code  reviews 

•  Implement  software  unit  testing 
°  Defect  Tracking  and  Reporting  System 

►  Regularly  review  and  improve  development 
and  test  methodologies 


Metrics 


►  Philosophy: 

°  NSA  Way  established  a  set  of  four  focus  areas  based  on 
key  business  objectives 

•  Time  to  Field 

•  Capabilities  Throughput 

•  Quality 

•  Efficiency 

►  Projects  define  their  own  criteria  and 
measurements  to  support  these  goals 

°  Gate  Pass/Fail  data  (Time  to  Field,  Throughput,  etc.) 

°  Code  Inspection  Data/Unit  Test  Data  (Quality) 

°  Customer  Satisfaction  Survey  (Quality) 

°  Defect  Data  (Quality) 

°  Program  Management  Data  (Efficiency) 


One  Year  Later... 


►  Accomplishments 
°  We  had  plenty 

►  Challenges 

°  OMG! 

►  Lessons  Learned 

°  You  bet 

►  Initial  Analysis 

The  Good,  the  Bad  and  the  Ugly 
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Accomplishments 


►  Overall  acceptance  of  the  framework 

°  Achieved  buy-in  at  all  levels:  Most  senior  level  to  grassroots 
°  More  than  double  the  number  of  projects  on  the  framework 
°  CMMI-like  processes  infused  without  using  the  ‘C’  word 

►  Creation  of  a  common  language 

°  Lifecycle  reference  points  for  managers  and  developers 
°  Expectation  of  common  processes  -  implementations  differ 
°  Improves  cross-project  comparisons  and  mobility 

►  Importance  of  the  ‘coaching’  model 

°  Coaching  model  becoming  a  tool  for  solving  other  types  of 
Agency  problems 

°  Less  “business  oriented”  than  a  Mentor,  broader  in  scope  than  a 
Guru  and  supported  by  a  coaching  network  and  infrastructure 

►  Acceptance  of  key  processes  and  gates 


Challenges 


►  Getting  good  data  and  getting  projects  to  use  it. 

►  Small  project  teams  challenged  by  ‘overhead’ 

►  Our  success  is  hindering  us  -  too  many  projects; 
too  few  coaches 

►  Balance  of  standardization  vs.  project’s  flexibility 
to  tailor  implementations 

►  Middle  management  acceptance 

►  Project  turnover 


Lessons  Learned 

►  ‘Handling  required’ 

►  Specialized  training  had  to  be  created 

►  Customer-facing  infrastructure  is  important 

►  Practice  what  we  preach 

►  Patience  goes  a  long  way 

°  Pressure  leads  to  Passive-Aggressive  behavior 

►  Tendency  to  never  report  failures 
°  If  it  ain’t  good,  don’t  report  it 


Analysis  of  Data 

►  Assessment  results 

►  Defect  analysis 

°  Cumulative  Open-Closed  DRs  over  Spins 

►  Gate  statistics 

Number  of  defects  by  gate  over  time 
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Initial  Assessment  Summary 
April  2009 


sfjATESOt 


NSA  Way  Product  Status 

Document 

External 

Interfaces 

Documented 

Coding 

Standards 

Hold  Code 
Reviews 

Implement 

Unit  Testing 

Documented 

CM  Processes 

Document  & 
Execute 
Defect 

Tracking  and 
Reporting 

Review  & 
Improve 
Development 
and  Test 
Methodologies 

Gate 

Operations 

Quality 

Metrics 

Final  Assessment  Summary 
June  201 0 
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NSA  Way  Product  Status 


Document  & 

Review  & 

Execute 

Improve 

Document 

Documented 

Defect 

Development 

External 

Coding 

Hold  Code 

Implement 

Documented 

Tracking  and 

and  Test 

Gate 

Quality 

Interfaces 

Standards 

Reviews 

Unit  Testing 

CM  Processes 

Reporting 

Methodologies 

Operations 

Metrics 

Number  of  DRs 


Defect  Analysis 


Months  in  a  Release 


•  •  • 


•X* 


R1  -  Open 
R1  -  Closed 
R2  -  open 
R2  -  Closed 
R3  -  Open 
R3  -  Closed 
R4  -  Open 
R4  -  Closed 
R5  -  Open 
R5  -  Closed 


Field 

Release 


Gate  Statistics 


Most  Occuring  Defects 
at  Initial  Review 


Gate  Delivery  VDD  with  RTM  Compatability  Open  DRs 
Date  Matrix  Documented 


Most  Occuring  Defects 
at  Final  Review 


■ 

No  Critical  Installation  VDD  with  RTM  Test 

Defects  Instructions  Procedures 

and  Results 
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Future  NSA  Way  Directions 


►  Complete  rollout  across  the  Technology 
Directorate  and  begin  expansion  beyond 

►  Encourage  teams  to  go  beyond  minimum 
expected  behaviors 

►  Improve  data  collection 

°  Customer  satisfaction 
°  Lots  of  other  directions  we  can  go... 

•  Advanced  requirements  and  defect  metrics 

•  Time  to  Market 

►  NSA  Way  2.0  (‘Raising  the  bar’) 

°  “Depth  and  Breadth” 

°  Integration  with  other  initiatives 

°  Learn  from  ‘our’  customers  and  continue  to  engage  the 
community 


Contact  Information 


►  Steve  Tobin,  Dynamics  Research  Corporation 

°  Email:  stobin@drc.com 

►  Michael  Mangieri,  Business  Transformation 
Institute,  Inc. 

°  Email:  mmangieri@biztransform.net 
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Integrated  Management  System 

“Leading  Our  Business  with  Purpose” 


Value  Based  Management  Establishes  Business  Strategies 


Value  Based  Product  Development  Transforms 
Strategies  Into  Product  Development  Tactics 

•  Stage  Gate 

-  Executive  direction 

-  Resource  allocation 

•  Nimba  Value  Model  focus  on  Customer  Value 

•  System  Engineering 

•  CMMI  best  practice  processes 
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Value  Based  Lean  Six  Sigma 

•  Measurement  and  Continuous  Improvement 


CMMI  Best  Practices  Execute  VBPDA/BM  Policies 
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Value  Based  Product  Development 
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ITT  Integrated  Management  System  (IMS) 


We  do 

essential  things  ^^B 
in  extraordinary  ways. 

Our  values  are  our  compass 
we  strive  to  do  the  light  thing  always 

Our  cusiomers  are  central 
to  everything  we  do  By  helping  them  reach  new 
levete  of  success,  vve  achieve  our  own 

Our  people  are  inspired 
to  make  the  world  a  better  place 
through  new  fcJea*. 

\  Our  work  h  critical  , 

to  create  more  livable  environments,  /a 
enable  commumcaborrs  and  /J 
provide  protection  j^gk 
and  safety 


^sPONSje/t/r>, 


Integrated  Strategic  Processes 


PROFITABLE 

RESOURCE 

OPERATIONAL 

LEADERSHIP 

<s*o 

OPTIMIZATION 

EXCELLENCE 

i  l  EARNING 

Value  Based 

Premier  Resource 

Value  Based 

Value  Based 

Management 

Management 

Lean  Six  Sigma 

Leadership 

Development 

Value  Based 

RortloJroAIapital 

Value  Based 

Product 

Allocation 

Goal 

Partnership  for 

Development 

Deployment 

Performance 
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IMS  Video  Screen  shots 


Power  of  IMS 
as  Integrated, 
Interdependent 
and  Iterative 
Framework 


Ik 


power  of  the  ITT  Management  System  is  when 
it  is  used  as  an  integrated,  interdependent  and 


Power  of  IMS 
as  Integrated, 
Interdependent 
and  Iterative 
Framework 


apart  is  that  we  embrace  the  use  of  the  ITT 
Management  System  in  a  disciplined  way 


with  purpose  -  based  on  facts  rather  than  gut  feelings  - 
and  guides  the  Premier  Resource  Management 


10151 


Nimba  Value  Arrow 
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CMMI  is  our  Framework  for  Product  Development 
Within  Our  IMS  Business  Framework 


IMS  Drives 
Process 
Measurement 
and 

Improvement 
Embedded  in 
VBLSS 


•  CMMI  for  Development  (CMMI-DEV)  provides  a 
comprehensive  integrated  solution  for  development  and 
maintenance  activities  applied  to  products  and  services 

•  CMMI®  (Capability  Maturity  Model®  Integration)  is  a  process 
improvement  maturity  model  for  the  development  of  products 
and  services. 
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Functions  Solutions  ,  Processes 


Value 


Value  Based  Product  Development 
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IMS  and  CMMI 


•  IMS  is  an  “...integrated,  interdependent  and  iterative 
framework,  guiding  our  strategic  decision  making,  resource 
allocation,  operational  excellence  and  leadership” 

•  CMMI  “processes  are  planned  and  executed  in  accordance 
with  policy;  the  projects  employ  skilled  people  who  have 
adequate  resources  to  produce  controlled  outputs” 

•  IMS  “...guides  what  we  do,  measures  how  well  we  execute, 
and  creates  options  for  doing  even  better” 

“...we  embrace  the  use  of  the  ITT  Management  System  in  a 
disciplined  way” 

•  CMMI  “...processes  are  well  characterized  and  understood, 
and  are  described  in  standards,  procedures,  tools,  and 
methods 


v>V 
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Functions  Solutions  ^Processes 


Value 


Value  Based  Product  Development 
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VBM  and  Strategic  Planning 
Focuses  Value  Based 
Product  Development  (VBPD) 


Strategic  Analysis 


Integrated  Strategic  Processes 


PROFITABLE 

GROWTH 


RESOURCE 

OPERATIONAL 

LEADERSHIP 

OPTIMIZATION 

EXCELLENCE 

&  LEARNING 

Premier  Resource 

Value-Based 

Value-Based 

Management 

Lean  Six  Siqma 

Leadership 

Development 

Portfolio  Capital 

Value-Based 

Allocation 

Goal 

Partnership  for 

Department 

Performance 

Mergers 

Acquisitions 

Divestitures 


Lasting  Value 
via  Corporate 
Portfolio 
Management 

Sustainable 

Profitable 

Growth 


VBM  Strategy  Development 


Develops  Product  Development 
Strategies  to  Address  Customer  Needs 
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VBPD  Uses  the  Nimba  Value  Model  to 
Focus  Product  Development  on  Customer  Value 


I 


Needs 


ml 


Functions 


Business  Systems 

Development  Engineering 


Solutions 

Design 

Engineering 


Processes 


Ml 


Manufacturing 

Engineering 


Functions  Satisfy  Solutions  Provide  Processes 
Customer  Needs  Functions  Produce  Solutions 


VBPD  Focuses  Products  and  Services  on  Customer  Value 
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Stage  Gate  Within  VPBD  Transforms  Strategy  into 
Tactics  to  Develop  the  Right  Projects 

'  ^  J  VBM 

to  Serve  Customer  Needs  Market  Segmentation/Selection 


Attractiveness  Competitive  Position 


Stage  Activities  develops  data  required  for  gate  decisions 


This  is  a  model  Stage-Gate  process  for  new  Defense  Product  Development.  It  is 
ITT  Defense  policy  that  each  Value  Center  adapts  a  Stage-Gate  tailored  for  their 
business. 

Defense  policies  imbedded  in  this  process  is  to  establish  a  process  that  incorporates 
“best-in-class”  practices  for  focusing  resources  on  finding,  qualifying,  pursuing  and 
winning  opportunities  by: 

•  Focusing  on  Gate  and  Milestone  decisions  and  the  information  required  for  the  decision 

•  Coupling  resource  allocation  and  expenditures  to  milestone  decisions 

^  ITT 


Needs 


Functions  Solutions  ,  Processes 


Value  Based  Product  Development 


Value 
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Engrg  BD 


Stage  Gate  Within  VPBD  Ensures 
We  Select  the  Right  Products 
For  Development 

Strategy 


VBM 

Market  Segmentation/Selection 
Attractiveness  Competitive  Position 


^  ITT 


Needs  Functions  Solutions  ^Processes  Value 


Value  Based  Product  Development 
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Apply  CM  Ml  Best  Practices  to  Develop  the  Project  Plan 


10151 


Nimba  Value  Arrow 
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Stage  Gate  Ensures  We  Apply  CMMI  Best 
Practices  to  Develop  the  Product  Right 


Strategy 


VBM 

Market  Segmentation/Selection 

Competitive  Position 

~r 


Attractiveness 


IR&D 

Technology  Development 


t 


Discover 


* 


ITT 


Idea 


Scope 


Propose, 
Win  and 
Plan 


Double  Click 
Links  to  Open 

PM 

Engrg 

Project  Monitoring  and  Control 
Supplier  Agreement  Management 
Integrated  Project  Management 

Risk  Management 

Integrated  Teaming 

Design  the  Product 

Document  the  design 

Design  comprehensive  product- 
component  interfaces 
Make/Buy/Reuse  Decisions 

Integrated  Supplier  Management 

Implement  the  Design 

Product  Support  Documentation 

Needs 


Functions  Solutions  ^Processes 


Value  Based  Product  Development 


Value 
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Processes  are  Tailored  to  the  Project 
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Needs  Functions  Solutions  Processes 
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Value  Based  Product  Development 


Value 
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v- j  - 


VBPD  VOC  to  Concept  Creation  and  Differentiation 

Through  Functional  Analysis 

^ - Validate  Needs  and  Deeper  Dive - 


Elicit  VOC 


L 


Identify  and  Prioritize  Customer  Needs 

i - 


Revise  or  Deeper  Decomposition 


Analyze 

Functions 


Identify  Functions  required  to  meet  Needs.  (What  the  system  should  do.) 

Develop  solution  Independent  Functional  Requirements  promoting  Innovation/Differentiation 


Validate 
Needs  & 
Deeper 
Dive 


Decompose  and 
Deeper  Dive 


Generate 

Concepts 


1 

1 

H 

i 

i 

i 

i 

i 

L 

A 

1 

*Spoken  & 

1 

1 

i  Unspoken 

1 

Analyze  and 

J  Non-Functional 

1 

| 

Select  Concept(s) 

Needs 

1 

1 

- Innovate/Differentiate  - 

Innovate 

Develop  Multiple  Concept  alternatives  to  provide  functions 


Correlate  Functions  and  performance  to  Customer  needs 
Assess  Competitively  (SWOT) 

Assess  best  customer  and  business  value  options 


Innovate 

I 


Team 

Review 


Loop  back  as  appropriate 


Refine  Selected 
Concept(s) 


Refine  concept  to  maximize  Customer  Value 
Identify  Constraints  and  Re-use 
Refine  business  case 


Develop  Design 
Strategy 


Plan  for  detailed  design. 

Establish  Functional,  Detail  Design, 
Performance  and  Cost  Requirements 


_  ,  ,  Gate  2  Requirements  Review 

Development  ◄ - <  >  ◄ - ,  - 

N/  Go  to  Gate  2  Loop  back  as  appropriate 
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Functions  Solutions  ^Processes 


Value  Based  Product  Development 
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VBPD  Kano  Analysis  of  Needs 


CMMI:  Eliciting  goes 
beyond  collecting 
requirements  by 

proactively  identifying 
additional 
requirements  not 
explicitly  provided  by 
customers 


CMMI:  The  basic 
activity  addresses 
the  receipt  of 
requirements  that  a 

customer  provides 
to  define  what  is 
needed  or  desired 


Satisfied 

(happy) 


Unspoken  Needs  and  Delighters 
that  customers  are  not  aware  of 
until  presented  to  them 

•  Differentiators  in  the  market 


Articulated  Needs 

All  competitors  strive 
to  meet  and  exceed 
requirements 


Performance 


Basic  Unspoken  Needs 

Basic  requirements  to 
enter  market 


r 


Elicit 

VOC 


—  Validate  Needs  &  Deeper  Dive 

Identify  and  Prioritize 
Customer  Needs 
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VBPD  Consider  All  Stakeholders 
in  the  Product  Life  Cycle 


CMMI:  ...  these 
requirements  address  the 
needs  of  relevant 
stakeholders,  including 
those  pertinent  to  various 
product  life-cycle  phases 

The  Customer  Value  focus  changes 
during  the  product  life  cycle. 

•  Prospects  are  attracted  with  image 

•  Customer  value  is  important  to  buyers 

•  Recipients  value  precision  in  accuracy  in  response  to  purchase  specifications 

•  User  values  quality  and  performance 

•  End  of  Life  Decommissioners  value  recycling  to  simplify  work 


Taken  from  “NIMBA  Value  Model”  Chapter  17  page  12 
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Needs 


Functions  Solutions  ,  Processes 


Value 
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VBPD:  Needs  are  Opportunities  Which  Should 
Be  Articulated: 


•  In  the  language  of  the  customer,  not  the  language  of  the 
designers 

•  In  the  domain  of  the  customer,  not  the  domain  of  the  product 

CMMI:  The  customer  requirements  may  be  expressed  in  the 
customer’s  terms  and  may  be  non-technical  descriptions. 


VBPD:  Independent  of  a  solution  (or  even  the  possibility  of  a  solution) 

•  As  a  “need  to”  ....(do  something);  not  as  a  “need  for” ...  something 

-  A  “need  for”  something  implies  a  solution 

•  Characterize  desired  outcome 

Validate  Needs  &  Deeper  Dive 

Elicit  Identify  and  Prioritize 
voc  Customer  Needs 


v>V 
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Needs  (  Functions  Solutions  Processes 


Value 


Value  Based  Product  Development 


10151 


Nimba  Value  Arrow 


17 


VBPD:  To  Ensure  Unrivaled  Customer  Value  in 
Product  Development,  Needs  Must  Be: 

•  Identified 

-  “You  can  observe  a  lot  just  by  watching.  ” 

•  Translated  into  clear,  concise  and  measurable  technical 
requirements 

-  “You’ve  got  to  be  very  careful  if  you  don’t  know  where  you  are  going 
because  you  might  not  get  there.  ” 


CMMI:  The  stakeholder  needs,  expectations,  constraints,  interfaces, 
operational  concepts,  and  product  concepts  are  analyzed,  harmonized, 
refined,  and  elaborated  for  translation  into  a  set  of  customer 

requirements.  s - 

Validate  Needs  &  Deeper  Dive 

I  Elicit  Identify  and  Prioritize 
voc  Customer  Needs 
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VBPD:  Needs,  Functions  and  Customer 
Value  Should  Drive  Concept  Selection 


“No  amount  of  architectural  or  detailed  design  rigor  can  make  up 
for  an  inferior  concept  selection ” 

CMMI:  Requirements  are  the  basis  for  design.  The  development  of 
requirements  includes  the  following  activities: 

•  Elicitation ,  analysis,  validation,  and  communication  of  customer 
needs,  expectations,  and  constraints  to  obtain  customer 
requirements  that  constitute  an  understanding  of  what  will  satisfy 
stakeholders 


Validate  Needs  &  Deeper  Dive 


Elicit  Identify  and  Prioritize 
voc  Customer  Needs 


Value 


Value  Based  Product  Development 
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Nimba  Value  Arrow 
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Functional  Analysis 


VBPD:  “to  capture  the  intended  behavior  of  the  system”... 
to  satisfy  the  customer’s  “need  to  do  something” 

CMMI:  The  definition  of  functionality,  also  referred  to  as 

“functional  analysis,”  is  the  description  of  what  the  product  is 
intended  to  do. 


f 


Validate  Needs  and  Deeper  Dive 


Elicit 

VOC 


Identify  and  Prioritize  Customer  Needs 


Revise  or  Deeper  Decomposition 


L 


f 


Analyze 

Functions 


Identify  Functions  required  to  meet  Needs.  (What  the  system  should  do.) 

Develop  solution  Independent  Functional  Requirements  promoting  Innovation/Differentiation 


Value 


Value  Based  Product  Development 


10151 


Nimba  Value  Arrow 


20 


Recursive  and  Iterative  Functional  Analysis  and  Concepts 
VBPD  Concept  Development 


Validate  Needs  &  Deeper  Dive 


Elicit 

VOC 


L 


•  Identify  and  Prioritize  Customer  Needs 

f - 


Revise  or  Deeper  Decomposition 


Validate 
Needs  & 
Deeper 
Div< 


Analyze 

Functions 


Identify  Functions  required  to  meet  Needs. (What  the  system  should  do.) 

Develop  solution  Independent  Functional  Requirements  promoting  Innovation/Differentiation 


Innovate/Differentiate 


Innovate 

Develop  Multiple  Concept  alternatives  to  provide  functions 


Correlate  Functions  and  performance  to  Customer  Needs 
Assess  Competitively  (SWOT) 

Assess  best  customer  and  business  value  options 


Analyze  and 
Select  Concept(s) 


Loop  back  as  appropriate 


Refine  Selected 
Concept(s) 


•  Ref ine  concept  to  maximize  Customer  Value 

•  Identify  constraints  and  Re-use 

•  Refine  business  case 


CMMI  Analyses  occur  recursively  at  successively  more  detailed  layers  of  a  product’s 
architecture  until  sufficient  detail  is  available  to  enable  detailed  design,  acquisition, 
and  testing  of  the  product  to  proceed.  As  a  result  of  the  analysis  of  requirements  and 
the  operational  concept ...  concept(s)  produces  more  derived  requirements 
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Value 
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Functional  Analysis  Drive  Solutions  and  Identifies 
Interface  Requirements 


VBPD 

Functional  Analysis 

...Moves  our  focus  from  outside  the  system  and  what  the  system  is  intended  to  do 
...to  inside  the  system  and  begin  to  specify  how  it  will  accomplish  its  intent 


CMMI 


Interfaces  between  functions  (or  between  objects)  are  identified.  Functional 
interfaces  may  drive  the  development  of  alternative  solutions 


^  ITT 


Needs 


Functions^^Solutions  i  *  Processes 
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Value 
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USE  Cases 


VBPD  ...  Study  the  desired  behavior  of  the  product  before  a 
solution  is  developed 

CMMI  ...Typical  Work  Products 

•  Timeline  analyses  of  product-component  interactions 

•  Use  cases 

Analyses  are  performed  to  determine  what  impact  the  intended 
operational  environment  will  have  on  the  ability  to  satisfy  the 
stakeholders’  needs,  expectations,  constraints,  and  interfaces. 


A 


* 


ITT 


Needs 


Functions  Solutions  ,  Processes 
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Value 
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Analyze  and  Validate  Requirements 
VBPD  Concept  Development 


Validate  Needs  &  Deeper  Dive 


Elicit 

VOC 


L 


1  Identify  and  Prioritize  Customer  Needs 

CT 


.Revise  or  Deeper  Decomposition 


I  Validate 
Needs  & 
Deeper 
Dive 


Analyze 

Functions 


Identify  Functions  required  to  meet  Needs.(What  the  system  should  do.) 

Develop  solution  Independent  Functional  Requirements  promoting  Innovation/Differentiation 

I -  Innovate/Differentiate  - 


Decompose  and 
Deeper  Dive 


Generate 

Concepts 


Innovate 

Develop  Multiple  Concept  alternatives  to  provide  functions 


*Spoken  & 
Unspoken 
Non-Functional 
Needs 


iL_ 


Analyze  and 
Select  Concept(s) 


Correlate  Functions  and  performance  to  Customer  Needs 
Assess  Competitively  (SWOT) 

Assess  best  customer  and  business  value  options 


Innovate 

I 


Team 

Review 


Loop  back  as  appropriate 


Refine  Selected 
Concept(s) 


Refine  concept  to  maximize  Customer  Value 
Identify  constraints  and  Re-use 
Refine  business  case 


CMMI  -  The  objectives  of  the  analyses  are  to  determine  candidate  requirements  for 
product  concepts  that  will  satisfy  stakeholder  needs,  expectations,  and  constraints;  and 
then  translate  these  concepts  into  requirements 


Requirements  are  validated  to  increase  the  probability  that  the  resulting  product  will 
perform  as  intended  in  the  use  environment. 
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Integrated  Management  System 

“Leading  Our  Business  with  Purpose” 


Value  Based  Management  Establishes  Business  Strategies 


Value  Based  Product  Development  Transforms 
Strategies  Into  Product  Development  Tactics 

•  Stage  Gate 

-  Executive  direction 

-  Resource  allocation 

•  Nimba  Value  Model  focus  on  Customer  Value 

•  System  Engineering 

•  CMMI  best  practice  processes 


* 


Value  Based  Lean  Six  Sigma 

•  Measurement  and  Continuous  Improvement 


CMMI  Best  Practices  Execute  VBPDA/BM  Policies 


v* 


<y 
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Agenda 


•  Context :  Problem  to  be  Solved 

•  Software  Architecture  Applicability  to 
Process  Improvement  Architecture 

•  Project  Phases 

•  Lockheed  Martin  Operating  Excellence 

•  Summary 
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Context :  Problem  Definition 


*  A  process-rich  culture  bred  many  process 
improvement  “entities”  over  a  decade+ 

-  Some  vintage  from  legacy  organizations 

-Some  spawned  to  respond  to  multiple 
models/standards  in  an  integrated  way 

-All  great  mechanisms  -  but  stovepiped! 


Project  Sponsored 


•  Sponsorship  and  initiation  of  a  process 
improvement  project: 

-  Objective:  Improve  the  Enterprise 
Process  Improvement  (PI)  Architecture 


November  17,  2010 


Defining  an  Enterprise  PI  Architectur 


•  What  comprises  an  enterprise  process 
improvement  architecture? 

-Approach  taken:  leverage  concepts  from 
software  architecture 


November  17,  2010 


Software  Architecture  Definition* 


•  The  structure  or  structures  of  the  system, 
which  comprise  the  software  elements, 

•  The  externally  visible  properties  of  those 
elements  and 

•  The  relationships  among  them* 


e  b  h  r* 


This  definition  can  be  applied  to  an  Enterprise 
Process  Improvement  Architecture 

*  Software  Engineering  Institute  Webinar  “Software  Architecture  Fundamentals: 
Technical,  Business,  and  Social  Influences”  by  Rob  Wojcik,  July  8,  2010 
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Enterprise  Process  Improvement — -A^ 
Architecture  Definition 

•  The  structure  or  structures  of  the  enterprise, 
which  comprise  the  process  improvement 
entities*, 

•  The  externally  visible  properties  of  those 
entities  and 

•  The  relationships  among  them 


*  Includes  people,  processes,  policies  and  tools 


Software  Architecture  Principles* 


•  Architecture  abstraction  includes  all  of  the 
“elements” 


•  Roles,  responsibilities,  behaviors  and  properties 
of  the  elements  are  described 


•  Relationships  depicted  include  those 
-Among  elements 

-  Between  elements  and  other  enterprise 
activities 

•  Artifacts  provided  to  and  required  from  elements 

•  Different  views  and  perspectives  available 

*  Also  apply  to  an  Enterprise  Process  Improvement 

Architecture 
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Process  Improvement  Architecture 

Elements 


The  Process  Improvement  Architecture  includes  everything  needed  to 

“operate”  process  improvement! 


November  17,  2010 


Project  Phases 


INVENTORY  IMPROVEMENT  ACTIVITIES 


INVESTIGATE  WHAT  OTHERS  ARE  DOING 


CREATE  FLOWS  OF  STATUS  QUO 


RELATE  TO  OPERATING 
EXCELLENCE 


LEAN  & 


ENGAGE  ENTERPRISE  STAKEHOLDERS 


COMPILE  FINDINGS  & 
RECOMMENDATIONS 


IMPLEMENT  RECOMMENDATIONS 
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Results  of  Process  Improvement 
Inventory 


•  Stovepiped  activities 

•  Unintegrated 

•  Not  all  are  institutionalized 

•  Nevertheless  . . . 

-Significant  process  improvement 
underway! 


An  Enterprise  Process 
Improvement  Architecture  has  no 


STOVEPIPES 


So  there  is  considerable  work  to  be  done! 
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Tenets  from  Other  Enterprise  Process^^ 
Improvement  Activities 

*  Focus  on  practices  resulting  in  measurable  business 
improvements 

*  Establish  a  process  improvement  framework 

-  GRIP  (Growth,  Reputation,  Innovation,  People) 

-  Protect/Transition/Grow 

*  Engage  Subject  Matter  Experts  from  each  business  unit 

*  Follow  a  business  rhythm  to  establish  initiatives  and  track 
progress 

*  Encourage,  but  don’t  edict,  best  practice  adoption 


Key  Practice:  Use  a  feedback  loop  to  guide  actions 
and  calibrate  whether  practices  are  working 
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Process  Innovation  &  Deployment  (PI&D) 
Process  Flow  -  Example 


PI&D  Lead: 

Enterprise  level 
leader  who 
maintains  the  PI&D 
Database,  PI&D 
Process,  and  PI&D 
Templates 

PI&D  POC:  One  Point 
of  Contact  from  each 
Product  Line  and 
Central  Function 


Award 

Project 
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Project  Phases 


INVENTORY  IMPROVEMENT  ACTIVITIES 


INVESTIGATE  WHAT  OTHERS  ARE  DOING 


CREATE  FLOWS  OF  STATUS  QUO 


:llence 


LEAN  & 


ENGAGE  ENTERPRISE  STAKEHOLDERS 


COMPILE  FINDINGS  & 
RECOMMENDATIONS 


IMPLEMENT  RECOMMENDATIONS 
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The  Path  to  Excellence  Uses... 
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Five  Principles  of  Operating  Excellence 

•  Specify  nm  of  the  product  from  your 
Customer’s  perspective 

•  Identify  the  2  W  P  J  lfor  each  product 

•  Enable  product  to  without 
interruptions 

•  Allow  the  customer  to  value  from  the 
producer 


•  Continuously  improve. ..pursue 


BILL  GEORGE 


r  1  i li  r  1  1  -  ■  Him- 


TRUE 

NORTH 
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Principle  Based  Approach 


Perfection 
/  Pull 

Flow 

Value  Stream 

Value  from  Customer's  Perspective 
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Eliminate  Non  Value  Elements 


•  Reduce  Waste 

•  Mistake  Proofing 

•  Reduce  Variation 


#  of 
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LM21  Operating  Excellence 

Logic  Map 


Leadership 
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Leader  Engagement 
Customer  Engagement 


LRP 


Op  Plan 


Productivity 
Targets  &  Strategy 


Quarterly 

Review 


Strategic 

Knowledge 

Discipline 

Communication 


Leadership  Council 
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Tactical 

>  Infrastructure 

>  Activity  and  Results 


Program 
Value  Streams 

Functional 

Program 

Review  -  Capture  -  Update 

&  Product 
Development 

- ► 

Excellence 

- ► 

Return  on  Invested  Capital 

Alignment 

Plans 

(ROIC) 

Kaizens  (PDKs) 

Improvement  Events 
(PDKs,  Kaizens,  Projects  JDIs) 


Closed-Loop  Continuous  Improvement 
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Project  Phases 


INVENTORY  IMPROVEMENT  ACTIVITIES 


INVESTIGATE  WHAT  OTHERS  ARE  DOING 


CREATE  FLOWS  OF  STATUS  QUO 


RELATE  TO  OPERATING 
EXCELLENCE 


LEAN  & 


ENGAGE  ENTERPRISE  STAKEHOLDERS 


Here  we  are! 


IMPLEMENT  RECOMMENDATIONS 
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Significant  Project  Findings 

•  Focusing  on  Operating  Excellence  at  the  enterprise  level 

-  Ensures  practices  result  in  measurable  business 
improvements 

-  Provides  a  business  rhythm  to  establish  initiatives  and 
track  progress 

-  Works  by  encouraging,  not  mandating,  best  practice 
adoption 

*  Directives  and  procedures  are  useful  to  flesh  out  the 

Operating  Excellence  process  (e.g.,  in  Product  Lines) 

-  Directives  and  procedures  mapped  to  the  corporate 
process  and  ensure  integration  and  completeness 

-  The  same  Operating  Excellence  process  applies  at  every 
level  of  the  corporation 
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Operating  Excellence  Deployment  at  Every  Fmrc 
and  Organizational  Level  in  the  Corporation 


IS&GS 

PRODUCT  LINES 


ENTERPRISE 
Lockheed  Martin 
Integrated 
Systems  & 
Global  Solutions 
(IS&GS)  _ 
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Summary 


•  Software  architecture  concepts  apply  to  the 
Enterprise  Process  Improvement 
Architecture 


-  Provide  discipline 

-Ensure  completeness 

•  Lockheed  Martin’s  Operating  Excellence  is  a 
fundamental  cornerstone  of  our  business 
model 


-Applies  to  process  improvement  activities 
at  every  level 

-  Provides  a  robust  tool  kit 
-Yields  business  results! 
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Contact  Information 


•  Joan  Weszka 

*  Lockheed  Martin  Information  Systems  &  Global 
Solutions 

-  301-721-5714 

-  Joan.Weszka@lmco.com 
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Using  CMMLto 
Improve  the  Enterprise, 
Not  Just  its  Processes 
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Philip  Wah,  Jr. 

The  Boeing  Company 
philip.b.wah-jr@boeing.com 
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Boeing  Space  and  Intelligence  Systems  (S&IS) 


S&IS  is  Boeing’s  center  for  all  intelligence  and  government  / 
commercial  space  systems.  It  is  the  world's  leading  manufacturer  of 
geostationary  satellites,.  and  has  the  largest  dedicated  satellite  factory 
in  the  world.  S&IS’  headquarters  is  in  Seal  Beach,  California. . 


CMMI®  Core  Team  Located  in  El  Segundo,  Southern  California 


ABSTRACT :  Using  CMMI®  to  Integrate  and  Improve  the 

Enterprise,  NOT  Just  its  Processes 


For  some,  the  CMMI®  Journey  is  about  defining,  executing  and 
improving  processes  with  the  end  goaj  of  obtaining  a  CMMI®  rating. 
Taking  this  approach  usually  results  with  good  processes,  satisfied 
process  teams,  a  plaque  on  the  wall  and  no  real  perceived  value. 

This  presentation  covers  how  we  made  high  value  achievements  when 
renewing  our.  CMMI®  rating.  This  included  making  significant 
organizational  integration  improvements,  and  fostering  a  culture  that 
recognizes  CMMI  makes  good  business  sense. 

The  key,  was  in  creating  value  propositions  using  the  right  people, 
use  cases  and  tools  to  facilitate  efficient  information  access.  Most 
importantly,  we  planned  big,  started  small  arid  had  a  detailed 
interpretation  and  implementation  strategy  of  the  CMMI®  model. 
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Agenda 


■  Background  -  Transitioned  to  strategic  CMMI®  focus 

■  High  leverage  areas  for  value  propositions 

-  Appraisal  preparation 

-  Multi-disciplined  problems 

-  Use  cases 

■  Critical  steps  in  framing  value  propositions 

■  Getting  MORE  value  from  CMMI® 


r - 

A  Value  Proposition  is  a  Proposal  that  Adds  Value 
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Where  Have  We  Been  on  Our  CM  Ml®  Journey? 


■  Started  using  CMMI®  as  an  improvement  tool  in  early  2000 

■  Obtained  numerous  CMM®  /  CMMI®  Level  3/5  ratings 

■  Received  numerous  RFPs  that  have  CMMI®  requirements 


Transitioned  focus  from  tactical  to  strategic  improvements 

-  Broad  scope 

-  Infrastructure 

-  Integration  ** 


«  Vu 


II 


r  >1 

Using  CMMI®  for  the  Right  Reasons  ! 
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CMMI®  Transition  Team  Included  all 
Product  Development  and  Process  Management  Stakeholders 


Proc  Mgmt 
Function 

Prog  Mgmt 
Function 

SW 

Domain  A 
_ 

SW 

Domain  B 


Business  Unit 
CMMI®  Leader* 


CM*  /  QA*  /  SMP  / 
Training 


Training 

Function 


CM 

Function 


QA 

Function 


Sup  Mgmt 


CMMI®  Tools* 


Engineering" 


Programs 


HW  Domain 
A 


HW  Domain 


CMMI®SMEs  /  Focus  area  Program  Execution 
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Global  CMMI®  Knowledge 
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Situation  During  our  CMMI®  Transition 


■  Very  challenging  business  environment 

-  Few  new  programs 

-  Extreme  competition 

-  Focus  on  bottom  line  results  -  overhead  cutbacks 


■  Significant  organizational  changes 

-  Restructuring  and  realignment 

-  New  common  processes 

-  Aggressive  improvement  initiatives 


/ - \ 

Climate  Focused  on  Adding  Value 
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Described  CMMI®  to  New  Management  Team 


Resources 


PEOPLE 


TECHNOLOGY  PROCESSES 


PRODUCE 

EFFECTIVE 


Seems  ambitious. 
What  value  are  we  gaining? 


Products  & 
Processes 


Plans 


Teaming 


Engineering  /  Support 


Status  /  Corrective  Action 


Process  Improvements 


(  ^ 

Global  Nature  of  Approach  Raised  Concerns 

v _ ! _ / 
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Succinctly  Communicated  Value  of  CMMI®  Transition 


■  Business  Case 

-  Customers  want  it,  competition  using  it 

-  Proven  best  practices  aligned  with  how  we  operate 

-  Supports  where  we  want  to  go 


■  Examples  of  Value  Propositions 

-  Shift  from  tactical  to  strategic  thinking 

-  Processes  that  are  easier  to  locate 

-  Better  integration  of  activities 

-  Faster  access  to  data 


OK,  but  let’s 
discuss  the 
next  ste 


Related  CMMI®  to  our  Business  and  Operations 
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Created  Value  Propositions  from 
Appraisal  Preparation 
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Appraisal  Preparation  Value  Proposition  1 : 
Enable  People  to  Understand  How  to  do  Their  Jobs 
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Appraisal  Preparation  Value  Proposition  2: 
Use  Data  Collection  to  Integrate  Critical  Activities 


How  can  we  better  leverage  our  training  efforts? 

Are  we  collecting  the  right  measures? 

How  can  we  better  focus  our  improvement  activities? 


O 


o 


1  1 

1 

Integrated 

Training 

Plans 

,  1  ~ 

i 

Integrated 

Measurement 

Plans 

i  i 

i 

Integrated 

Improvement 

Plans 
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Numerous  Value  Propositions 
Stemmed  from  Multi-disciplined  Problems 


Solution 


Problem 

■  Evaluation  data  not  integrated 

-  Remove  redundancies 

-  Ensure  comprehensiveness 


Leverage  CM  Ml® 
PPQA  &  GP  2.9 


INTEGRATED  EVALUATION  ACTIVITY 

Internal  Programs  Functions  Corporate  Customers  AS9100  CMMI 


Process  A 

Future 

Progs  1  -  5 
(2008) 

nu  nnnn\  oc 

Recant 

2008  Jan 

Oust  1  (2009) 

2007  Aug 

Progs  1-3 
(2009) 

Process.  B 

2008  Feb 

Prog  6  (2008) 

Future 

Oust  3  (2010) 

Future 

Progs  3-4 
(2009) 

Process  C 

Future 

Future 

Test  (2009) 

2010  Oct 

2009  Aug 

Progs  2-4 
(2009) 

Process  D 

Future 

Future 

Future 

Future 

Progs  2-4 
(2009) 

Process  E 

Redundant 

zuiu  reu 

Prog  8  (2008) 

Future 

Oust  2  (2010) 

Future 

Progs  4-5 
(2009) 
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Other  Value  Propositions 
Based  on  Simple,  High  Priority  Use  Cases 


“Value  Added  !!” 


Target  Users 

Use 

Case 

# 

Use  Case  Examples 

Prog,  Func 
Practitioner 

Process 

Councils 

Process 

SME 

CMMI® 

SME 

Customer 

X 

X 

X 

1 

/ - 

Are  the  right  processes  being  deployed? 

X 

X 

X 

X 

2 

How  do  1  do  my  job? 

V _ y 

X 

X 

X 

X 

3 

/ - > 

Where  can  1  find  process  related  data? 

X 

X 

X 

X 

4 

/ - ' 

How  effective  is  our  evaluation  program? 

X 

X 

5 

What  is  our  CMMI®  approach? 

s. _ J 
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Framing  Complex  Value  Propositions 


■  Communication  Challenge 

-  CMMI®  terminology 

-  Broad  scope  of  CMMI®  effort 


■  Solution 

1 .  Translate  CMMI®  into  how  we  operate 

2.  Focus  audience  on  the  value  proposition’s  scope 

3.  Illustrate  the  value  of  the  proposition 
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Step  1  -  Translate  CMMI®: 

Process  Management  in  Our  Own  Paradigm 


Training  &  Measurement 
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Step  2  -  Focus  the  Audience: 

Large  Quantity  of  Process  Management  Data 
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Step  3  -  Illustrate  the  Value: 

Turn  Chaos  into  Organized  Process  Asset  Library  Views 


^  l 

■ 

Guides 

Improvement 

SharePoints 


Improvement 

Servers 


Process  Framework 


Process  Views 


Standard  Tools 


Trainina  Plans 


Process  Audits 


Measurement  Plans 


Measurement  Repositories 


Improvement  Plans 


t  Library 
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CMMI®  Accelerated  Our  Capabilities  in  a 
Challenging  Environment 


■  Integrated  new  processes 

-  “What  to”,  “How  to”  and  “When  to”  execute 

■  Defined  clear  RAAs  for  operating  teams 

■  Integrated  training,  measurement  and  improvement  plans 

■  Provided  single  portal  with  global  access  to  process  data 

■  Implemented  comprehensive  data  management  approach 

■  Consolidated  process  and  product  evaluation  information 


All  Validated  by  a  CMMI®  Level  3  Rating 
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Getting  MORE  Value  from  CMMI® 


■  Develop  detailed  CMMI®  implementation  strategy 

■  Involve  all  stakeholders 

■  Understand  high  leverage  areas  for  value  propositions 

■  Frame  your  value  propositions  as  simply  as  possible 

-  Translate  CMMI®  using  operational  scenarios 

-  Focus  discussions  to  the  audience 


Use  KISS:  Keep  It  Simple  Strategy  !! 
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Creating  CMMI  Value  Propositions 


Questions? 
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Raytheon  Missile  Systems’  CMMI®  Environment 


Raytheon 

Missile  Systems 


■  Raytheon  is  comprised  of  multiple  businesses 

-  One  business  is  Raytheon  Missile  Systems  (MS) 

•  Characterized  by  high-(defense)  volume,  complex  systems--“Rocket  Science” 

•  12,000  people,  >5000  engineers 

-  Each  Raytheon  business  manages  its  own  CMMI®  goals  and 
appraisals 

•  Knowledge  sharing  across  business  units 

•  Prior  Raytheon  and  RMS  experience  with  SW-CMM,  CMMI  appraisals 

-  Missile  Systems  has  been  appraised  at  Maturity  Level  5  for  CMMI  for 
Development  version  1.2  +IPPD 
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Does  your  organization  do  this? 
[We  did] 


Raytheon 

Missile  Systems 


Evidence  Collection 

-  Collect  something  to  cover  everything  possible  in  the  CMMI  model? 

-  Address  each  and  every  sub-practice? 

-  Provide  an  example  for  each  discipline  in  your  organization? 

-  Collect  evidence  even  when  a  demonstration  or  interview  would  more 
readily  provide  the  evidence  or  affirmation? 


Team  Behavior 

-  Bow  to  the  understandings  and  “expectations”  of  the  appraisal  team, 
even  then  they  differ  from,  or  are  more  restrictive  than,  the  CMMI  model? 

Model  Interpretation  ’A 

-  If  a  compound  statement  (such  as  GP3.2)  is  given  in  the  CMMI,  then 
provide  evidence  to  address  every  element  of  that  compound  statement? 

-  If  something  is  good,  then  providing  more  of  it  must  be  better? 
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Evidence  Collection-1 
“A  Thread  for  All  Disciplines” 


Raytheon 

Missile  Systems 


We  used  to  provide  evidence  threads  for  all  disciplines. .  . 

■  Decision  Analysis  and  Resolution  [DAR]  examples  shown 

-  Software  Thread 

[It  must  be  good  because  we  passed  Sl/l/  CMM  with  it!] 

•  E.g.,  software  DAR  -  which  board  support  package  vendor  to  use 

-  Hardware  Thread 

[Yeah,  more  than  just  SW... let’s  add  a  HW  example  tool] 

•  E.g.,  hardware  DAR  -  microprocessor  choice 

-  Systems  Engineering  Thread 

[Let’s  combine  SW  and  HW,  and  show  some  tradeoffs  and  “big  picture 

•  E.g.,  decision  to  do  hardware  or  software  filtering  of  images 

-  “Glue-ware”  and  others 

[Hmm,  we’ve  got  more  stuff  than  SW,  HW,  &  SysEng,  what  do  we  do  with  that?] 

•  To  explain  how  the  choices  are  related  to  a  program  or  how  they  connect 

•  E.g.  a  make  or  buy  decision,  where  to  rent  a  test  facility,  etc. 

■  We  probably  don’t  need  all  these 
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Evidence  Collection  -  2 
Generic  Practice  “Bingo” 


Raytheon 
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■  When  generic  practices  [GP’s]  contain  compound 
statements,  we  used  to  fill  out  the  entire  “bingo  card” 

-  GP  “Tsar’s”  were  appointed  to  ensure  completeness  of  each  GP’s 
“bingo  card” 

•  We  gathered  examples  to  address  as  many  permutations  as  possible 

-  Examples 

•  GP  2.3  Provide  Resources 


□  Schedule,  Budget,  People,  Tools,  Facilities,  Resources 
•  GP  3.2  Collect  Improvement  Information 


□  Work  Products,  Measures,  Measurement  Results,  Improvement 
Information 


“It  would  be  curious  if .. .  *” 

□  We  didn’t  have  anything  in 
some  rows  or  columns,  but  we 
don’t  need  a  full  “Bingo  Card” 


PA 

Excal 

JSOW 

TTL 

KA 

NLOS 

CM  GP3.2 

1 

1 

1 

1 

DAR  GP3.2 

1 

1 

1 

IPMSP1.5 

1 

1 

1 

1 

PI  GP3.2 

1 

1 

1 

PMCGP3.2 

1 

1 

1 

1 

PPQAGP3.2 

1 

SAM  GP3.2 

1 

1 

1 

TS  GP3.2 

1 

1 

1 

VAL  GP3.2 

1 

1 

1 

VERGP3.2 

1 

1 

1 

*  Thanks  to  JFR  Consulting  for  the  use  of  this  phrase 


11/17/2010  Page  6 


Evidence  Collection  -  3 
Putting  it  All  Together  (yikes!) 


Raytheon 
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3  *  4  *  6  =  72  pieces  of  evidence  for  each  GP2.3! 


Programs: 

-  A 

-  B 

-  c 


Disciplines: 

-  Hardware 

-  Software 

-  Systems  Engineering 

-  [Radar,  Optics,  Propulsion,  Guidance...] 

GP2.3  elements: 

•  Schedule 

•  Budget 

•  People 

•  Tools 

•  Facilities 

•  Resources 
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Evidence  Collection  -  4 
More  Fun  with  Generic  Practices 


Raytheon 

Missile  Systems 


■  GP  2.2  Plan  the  Process 

-  Establish  and  maintain  the  plan  for  performing  <x>  process 
•  From  page  620  of  CMMI  vl  .2  for  Development  (glossary): 


□  [“Establish  and  Maintain”]  means  more  than  a  combination  of  its  component  terms;  it 
includes  documentation  and  usage.  For  example,  “Establish  and  maintain  an 
organizational  policy  for  planning  and  performing  the  organizational  process  focus 
process”  means  that  not  only  must  a  policy  be  formulated,  but  it  must  be  documented 
and  it  must  be  used  throughout  the  organization.” 


•  Some  appraisal  teams  insist  on  finding  original  release  of  plan,  plus  revision 
to  the  plan 

•  Alternatively,  references  to  the  plan;  interview  citations;  other  indications  of 
use  (for  instance,  peer  review  references  to  the  plan  or  of  the  plan)  also  show 
maintenance 

•  We  may  not  need  that  “extra”  piece  of  “maintain”  evidence 
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Evidence  Collection  -  5 
Shotgun  Approach  to  Evidence 


Raytheon 
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■  Hmm,  I’m  not  sure  how  well  this  addresses  the  practice,  so 

■  I’ll  provide  multiple  examples...  in  fact,  I’ll  provide 

■  An  example  for  each  subpractice 

■  And  if  I’m  not  sure,  then  I’ll  provide  multiple  examples 
for  each  subpractice,  because 

■  Each  one  should  cover  a  little  of  the  practice,  and 

■  All  together,  they  should  be  good,  and 

■  The  appraisal  team  will  pick  the  ones  they  like  best! 

KA-BOOM! 

(Somebody  in  this  room  once  provided  13  examples  for  a  single  practice.) 
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Evidence  Collection  -  6 
Collect  Evidence  We  Don’t  Need 


Raytheon 

Missile  Systems 


■  We  can  collect  indirect,  even  though  we  expect  we  will  get 
affirmations  during  interviews — let’s  be  sure 

■  We  plan  to  hold  a  demonstration,  but  let’s  collect  screenshots  of 
the  demonstration  content  anyway...  you  never  know 


-  If  we  had  a  little  more  evidence  in  this  area 

•  Because  most  of  our  evidence  has  three  or  four  and 
examples;  this  one  only  has  two  (that  meet  the  model),  or 

-  If  we  had  more  kinds  of  examples,  or 

-  If  we  had  the  exact  same  kind  of  evidence  from 
each  program,  or 

•  If  we  had  some  of  the  programs  re-work  their  evidence 
to  match  the  format  of  the  other  programs,  or 


■  The  evidence  in  this  practice  complies  with  the  CMMI,  but  I  would 
feel  better.  .  . 


-  If  we  could  send  a  message  to  the  program  that  they  need  to  change 
because  I  don’t  like  how  they  do  it... even  though  I  guess  it  does  meet 
CMMI.  But  I  don’t  like  it.  It’s  not  what  I’m  used  to  seeing. 
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Team  Behavior  - 1 

True  Statements  from  Team  Members 
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■  Let’s  write  this  up  to  change  the  program/organization 
behavior  because  even  though  it  complies  to  the  model,  they 
need  to  change 

■  If  one  thread  is  good,  then  three  should  be  better 

■  If  we  are  uncertain  about  a  particular  process  area,  we  can 
provide  more  examples 

■  If  some  evidence  is  good,  then  more  is  better 

■  What  I  expect  to  see  is... 
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Team  Behavior  -  2 
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What  I  Remember  (or  Am  Used  To)  Must  Be  True 

■  Do  everything  that  the  Appraisal  Team  Lead  wants — 
unquestioningly 

-  The  Lead  Appraiser  has  one  (albeit  important)  vote  on  the  team 

■  Do  everything  that  each  appraisal  team  member  wants  or 
expects 

-  We  have  to  do  this — we’ve  always  done  it  this  way  in  our  other 
appraisals 

-  The  CMMI  model  is  a  wonderful  resource  -  consult  it  and  the  Method 
Definition  Document  [MDD]  often 

-  Maybe  it  doesn’t  say  quite  what  you  remember 
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Model  Interpretation  - 1 

Ways  to  Focus  on  the  Wrong  Things 
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■  We  need  to  get  our  CMMI  certificate,  so.  .  . 

-  We  are  going  to  drive  changes  to  the  program  to  support  the 
quantitative  processes  we  have  selected 

•  We  will  disrupt  and  perturb  programs,  but  we  will  get  our  certificate,  so  that’s 
OK--lt  doesn’t  matter  whether  the  changes  “stick” 

Versus 

■  We  need  to  help  our  organization  improve,  so.  .  . 

-  Let’s  find  what  our  programs  and  organizations  need  to 
focus  on  to  meet  their  goals 

-  Let’s  use  our  Six  Sigma  and  CMMI  model  tools  and  best 
practices  to  support  meeting  those  goals 

•  Programs  are  satisfied  and  will  agree  to  work  with  us  again 

•  Programs  keep  new  behaviors  that  helped  them  address 
their  risks 
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Model  Interpretation  -  2 
If  Some  is  Good,  All  is  Great 


Raytheon 
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Measurement  and  Analysis  Process  Area 

-  MA  can  be  used  for  all  other  process  areas 

-  Therefore,  it  should  be  used  for  all  process  areas,  so 

-  We  are  going  to  invent  at  least  one  metric  that  will 
measure  something  about  each  process  area,  and 

-  We  will  mandate  them  to  every  project 

-  Then  we  will  improve  everything! 

•  (At  least  we  will  have  a  giant  database  of  metrics,  and  we  can  figure  out  later 
what  is  most  important) 
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6 
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216 

. 

Mill . .  1  1  1 
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1  II  1  1  1  1 

1  I  1  I  1  I  1  I  1  I  1  I  1  I  1  I  1  I  1 

10%  20  30  40  90  60  70  80  90 

■  I  ■  I  ■  I  ■  I  ■  I  ■  I  ■  I  ■  I  ■  I  ■ 


Ooops! 

-  Let’s  think  about  which  are  the  key  factors  that  affect  our  performance 
to  our  goals,  and  prioritize  measurement  and  analysis  of  those 


A  selective  application  of  MA  can  make  sense  in  many  organizations 

A  metric  of  “DAR’s  planned  versus  DAR’s  executed”  may  not  be  the  best  way 
to  expend  resources 


CDF  Run  il  Preliminary  4. 


J  ■: 


11/17/2010  Page  14 


Summary 
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■  Barriers  to  efficient,  effective  appraisals  include 


-  Misguided  evidence  collection 

-  Certain  team  behaviors 

-  CMMI  model  misinterpretations 
■  Ways  to  avoid  these  barriers 

-  Live  through  these  mistakes 
•  And  learn  from  them 


Learn  from  somebody  else’s  mistakes 
Be  brave  about  using  common  sense 
Make  the  project  and  organizational  goals  your  guiding  principles 


•  Don’t  think  about  things  from  the  CMMI  model/appraiser  perspective  first 

•  The  programs  and  organizations  are  your  customer  and  priority 


■  Cautions 


-  Sometimes  you  do  need  multiple  pieces  of  evidence  or  threads 

-  Sometimes  you  should  listen  to  your  appraisal  lead!  © 
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Questions? 
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Presenter  Biography 
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27  years 
engineering 
experience 


Functions:  software,  systems  engineering,  program  management  liaison,  process  improvements 
Industries:  cell  phones,  industrial  automation,  missile  systems 
•  3  patents  -  high-reliability  systems,  redundancy,  inter-process  communication 
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Abstract 
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■  Abstract  ID:  10779 

-  Title:  CMMI-The  Way  It  Should  Be 

■  Abstract  Text:  CMMI-The  Way  It  Should  Be 

■  Why  have  three  times  as  many  artifacts  as  necessary  when  doing  a  CMMI  appraisal?  Why  complicate  the  data 
collection  and  appraisal  process  by  providing  excess  information  and  elaboration? 

■  Many  organizations  have  evolved  their  approaches  to  CMMI  compliance  by  building  on  prior  work  in  the  CMM 
and  CMMI.  Too  often,  these  approaches  focus  on  software  as  the  lead  thread,  and  supplement  this  with 
hardware,  systems  engineering,  and  disciplinary  threads.  Frequently  these  approaches  represent  an 
amalgamation  of  early  successes  and  lessons  learned  that  have  been  cobbled  together  to  cover  all  the  CMMI 
process  areas.  Sometimes  they  take  the  approach  that  if  something  from  one  discipline  is  good,  it  needs  to  be 
replicated  in  all  disciplines.  This  approach  can  result  in  redundant  or  less-value-added  effort  to  the  appraised 
organization. 

■  Why  not  instead  begin  with  a  truly  integrated  approach  that  does  not  segregate  or  isolate  disciplines?  (After  all, 
“Integration”  is  the  “I”  in  “CMMI”!)  The  author  proffers  that  using  an  integrated  approach  as  a  starting  point  for 
improving  using  the  CMMI  as  framework  saves  cost  and  time,  simplifies  data  collection  and  the  actual  appraisal 
process,  and  does  not  increase  the  risk  to  the  organization. 

■  Examples  from  our  organization’s  recent  CMMI  Level  5  appraisal  will  be  cited  to  show  the  differences  between 
the  Raytheon  Missile  Systems  approach  and  the  approach  we  have  seen  presented  by  many  others.  In 
particular,  aspects  of  data  collection,  data  and  thread  coverage,  and  ways  to  meet  the  requirements  of  the  CMMI 
and  the  Method  Description  Document  (MDD)  will  be  addressed.  The  presentation  will  also  discuss  training  and 
“big  picture”  views  of  this  approach  to  CMMI  and  appraisals. 
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Small  Business  in  the  Defense  Industry 


STRENGTH  T1IK01GII INDUSTRY  &  TECHNOLOGY 


The  defense  industry  is  critically  dependent  on  small  business  for  the 
technology  and  rapid  innovation  needed  for  future  defense  business. 

“...In  one  of  the  few  studies  of  its  kind,  the  Pentagon’s  industrial  policy  office  in  2005  identified 
the  critical,  innovative  technologies  which  are  needed  for  21st  century  wars  and  found  that  35 
to  45  percent  of  the  companies  supplying  those  key  technologies  had  fewer  than  100 
employees.” 

-  ND/A  National  Defense  Magazine  (June  2010), 

http:/ /www.  nationaldefensemagazine.  org/archive/20 1_  O/June/Pages/PentagonNeedsinnovativeMid-  TierContractors.  aspx 

However,  small  businesses  can  be  challenged  to  compete  and  thrive 
in  the  defense  market. 

"...  Only  the  largest  firms  have  access  to  the  resources  and  expertise  to  bid  on  the  most 
complex  programs,  and  it  is  difficult  for  all  but  the  largest  size  to  survive  losing  them.  As  a 
result,  competition  is  reduced  at  the  front  end  of  programs,  and  all  but  eliminated  in  the 
sustainment  phase....  Small  businesses  are  largely  locked  out  of  the  process  or  accorded 
contracts  only  on  the  goodwill  of  one  of  the  larger  firms....  The  end  result  of  this  process  is 
the  gradual  erosion  of  competition  and  innovation  in  the  defense  industrial  base.” 

-House  Armed  Services  Committee  Panel  on  Defense  Acquisition  Reform ,  Findings  and  Recommendations  (March  2010) 
http://armedservices.house.aov/Ddfs/DARFINALREPORT/DARFINALREPORT03231Q.Ddf 


CMMI®  for  Small  Business 
NDIA  CMMI  Working  Group 


2 


Is  CMMI  for  Small  Business? 


STRENGTH  TIIKOIGII 1NDLSTRV  &  TECHNOLOGY 


The  NDIA  CMMI  Working  Group  was 
tasked  to  investigate  common  questions 
on  the  applicability  of  CMMI  to  small 
business 

•  Is  CMMI  appropriate  for  small  businesses? 

•  Is  CMMI  a  barrier  to  competition  in  the 
defense  market? 

•  What  data  substantiates  the  adoption  of 
CMMI  by  small  business? 

•  What  challenges  do  small  businesses  face 
for  adoption? 

•  How  can  acquirers  use  CMMI  to  manage 
their  acquisition  risks  with  small  suppliers? 
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What  is  a  Small  Business? 


STRENGTH  T1IK01GII INDLSTRV  &  TECHNOLOGY 


There  is  no  single  definition  of  what  constitutes  a  small  business. 

The  SBA  characterizes  small  businesses 
in  terms  of  size  and  revenue,  by  domain 

•  Size  standards  defined  by  North  American  Industrial 
Classification  System  (NAICS) 

•  Typically  -250  employees,  <  $25M  in  revenue 

SEI  research  characterizes  a  small  setting  as: 

•  Small  business  <  100  people 

•  Small  organization  within  a  large  organization,  <  50  people 

•  Small  project  <  20  people 

For  this  study,  we  did  not  set  thresholds,  but  focused 
on  employee  size  across  a  wide  range  of  data  partitions 

•  As  small  as  a  few  people 

•  As  large  as  several  thousand  people 

•  ‘Small’  may  be  in  the  eye  of  the  beholder 


NDIA  Small  Business  Division 

http://www.ndia.org/Divisions/Divisions/SmallBusiness/ 

Paqes/default.aspx 


U.S.  Small  Business  Administration  size  standards: 
http://www.sba.gov/contractinaopportunities/officials/size/index.html 
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Prior  Studies  and  Research 


STRENGTH  THROUGH INDUSTRY  &  TECHNOLOGY 


Many  prior  initiatives  have  investigated  CMMI  for  small  business. 
Example  resources  include: 


CMMI  in  Small  Settings  Toolkit  Repository 

Suzanne  Garcia  (SEI),  Sandra  Cepeda  (AMRDEC  SED/CSSA),  Gene  Miluk 
(SEI),  Mary  Jo  Staley  (AMRDEC  SEC/CSC) 
http://seir.sei.cmu.edu/toolkit/index.html 


Improving  Processes  in  Small  Settings  (IPSS) 

International  Process  Research  Consortium  (IPRC) 

“Proceedings  of  the  1st  International  Researcher’s  Workshop  on  Process 
Improvement  in  Small  Settings” 

http://www.sei.cmu.edu/publications/documents/06.reports/06sr001.html 


“An  exploratory  study  investigating  the  organizational  and 
technical  impacts  of  applying  disciplined  system  development 
processes  (CMMI™)  in  small  to  medium  sized  enterprises.” 
Miluk,  Gene.  Ph.D.  dissertation,  Pepperdine  University,  2006. 
http://qradworks.umi.com/32/24/3224161.html 


The  CMMI  WG  conducted  interviews  and  discussions  with  several 
of  these  researchers  and  subject  matter  experts. 
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CMMI  for  Small  Business  - 

Data  Collection  and  Analysis 


STRENGTH  T1IK01GII  INDLSTRV  &  TECHNOLOGY 


There  are  plenty  of  anecdotes,  speculations,  and  opinions  on  the 
applicability  of  CMMI  for  small  business  -  what  does  the  data  tell  us? 

•  SEI  CMMI®  Process  Maturity  Profile*  Cmjm* 

http://www.sei.cnnu.edu/cmnni/casestudies/profiles/cmnni.cfm 


•  SEI  Published  SCAMPISM  Appraisal  Results  (PARS)* 

http://sas.sei.cmu.edu/pars/ 

•  Organizational  change  research 

Dr.  Leigh  Riley,  leiqh@acmeprocessqroup.com 

•  1st  International  Research  Workshop  for  Process 
Improvement  in  Small  Settings 

http://www.sei.cmu.edu/librarv/abstracts/reports/06sr001.cfm 

•  Surveys,  questionnaires,  ... 


*Caveats:  Most  CMMI  data  is  reported  and  collected  from  appraisals  based  on  size  of  the  organizational  unit 
-  which  may  or  may  not  be  directly  reflective  of  the  size  of  the  company  overall. 

Other  companies  may  also  be  using  CMMI  beyond  just  those  conducting  and  reporting  SCAMPI-A  appraisals. 
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CMMI  Adoption 


STRENGTH  THROUGH INDUSTRY  &  TECHNOLOGY 


CMMI 
appraisals 
are  conducted 
worldwide... 


USA  Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

425 

35.2% 

3354 

91.8% 

651 

54.0% 

223 

6.1% 

130 

10.8% 

76 

2.1% 

1206 

100.0% 

3653 

100.0% 

...in  small 
and  large 
organizations 
and  projects 


>200,  J 

22.4%  [ 

101-  f 
200,  1 
17.3%  r 

<100, 

60.2%  < 


>2000,1.6% 
1001-2000,3.3% 
501-1000,  5.0% 
301-500,  6.1% 
201-300,  7.3% 

101-200,  17.3% 
76-100,  8.3% 
51-75,  12.6% 


26-50,  22.0% 


<25,  17.3% 


25% 

...in  a  wide  range  of  businesses 


75% 


Organization  Size  (Employees) 

(4863  organizations  reporting) 


...at  all  levels  of  process  maturity 


Services  (72.1%) 

•  Business  Services 

•  Engineering  and 
Management  Services 

•  Health  Services 

•  Other  Services 


Based  on  primary  Standard  Industrial 
Classification  (SIC)  codes  reported 
in  CMMI-based  appraisals. 


Manufacturing  (15.7%) 

•  Electronic  and  Electric  Equipt 

•  Transportation  Equipment 

•  Instruments  &  Related 
Products 

•  Industrial  Machinery 

•  Other  Mfg  Industries 

Other  (12.2%) 

•  Finance,  Insurance,  Real  Estate 

•  Public  Administration/Defense 

•  Transportation,  Communication, 
Utilities 


Contractor  Military/ 


Commercial  for  Military/  Government 
In-House  Government  Agency 


No  Rating  Given 

5.3% 

8.0% 

22.3% 

Initial  (ML1) 

0.6% 

1.4% 

1.0% 

Managed  (ML2) 

25.8% 

31.5% 

45.6% 

Defined  (ML3) 

58.1% 

49.3% 

26.7% 

Quantitatively  Managed  (ML4) 

2.9% 

1.0% 

1.5% 

Optimizing  (ML5) 

7.2% 

8.9% 

2.9% 

(3779  orgs) 

(874  orgs) 

(206  orgs) 

Source:  SEI  Process  Maturity  Profile,  Sept  2010. 
http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 


CMMI  for  Executives 
November  2010 


Reference:  ‘CMMI  for  Executives’,  NDIA  CMMI  Working  Group,  CMMI  Technology  Conference  and  User  Group,  November  2010. 
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Adoption  Trends  in  Small  Business 

Comparison:  SEI  Process  Maturity  Profile,  Sep  2009  vs.  Sep  2010 
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Sep  2009 


>200, 

25.2% 


>2000,1.9% 
1001-2000,3.3% 
501-1000,  5.6% 
301-500,  6.7% 
201-300,  8.0% 

101-200,  18.9% 
76-100,  8.3% 
51-75,  12.8% 

26-50,  19.9% 
<25, 15.0% 


^tion  Size  (Employees) 
(3863  o  rgato  izati  ons  rep  orti  ng) 


Commercial 

In-House 

Contractor 
for  Military/ 
Government 

Military/ 

Government 

Agency 

No  Rating  Given 

5.7% 

8.5% 

22.7% 

Initial  (ML1) 

0.8% 

1.6% 

1.7% 

Managed  (ML2) 

28.0% 

31.7% 

44.3% 

Defined  (ML3) 

53.6% 

46.7% 

26.7% 

Quantitatively  Managed  (ML4) 

3.1% 

1.4% 

1.1% 

Optimizing  (ML5) 

^ - 

10.1% 

3.4% 

(^920  orgs^ 

(769  orgs) 

(176  orgs) 

USA 


Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

354 

33.6% 

2566 

91.3% 

586 

55.7% 

183 

6.5% 

113 

10.7% 

63 

2.2% 

1053 

100.0% 

2812 

100.0% 

27.2% _ C/72.8%^) 
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Sep  2010 

Greatest 
growth  in  CMMI 
adoption 
worldwide 
continues  to  be 

•  small 

•  commercial 

•  overseas 


>200, 

22.4% 


(  60.2% 


>2000, 1.6% 
1001-2000,3.3% 
501-1000,  5.0% 
301-500,  6.1% 
201-300,  7.3% 

101-200,  17.3% 

76-100,  8.3% 

75,  12.6% 


itjon  Size  (Employees) 

(4863  organtations  reporting) 


No  Rating  Given 
Initial  (ML1) 
Managed  (ML2) 
Defined  (ML3) 
Quantitatively  Managed  (ML4) 
Optimizing  (ML5) 


Commercial 

In-House 

Contractor 
for  Military/ 
Government 

Military/ 

Government 

Agency 

5.3% 

8.0% 

22.3% 

0.6% 

1.4% 

1.0% 

25.8% 

31.5% 

45.6% 

58.1% 

49.3% 

26.7% 

2.9% 

1.0% 

1.5% 

_  7.2% 

8.9% 

2.9% 

r  (3779  orgsft 

(874  orgs) 

(206  orgs) 

USA 


Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

425 

35.2% 

3354 

91.8% 

651 

54.0% 

223 

6.1% 

130 

10.8% 

76 

2.1% 

1206 

100.0% 

3653 

100.0% 

24.8% 


Q.2°/Q 


What’s  the  CMMI  maturity  profile  for 
small  businesses? 
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C 

O 


O 

H— 

O 


CarnegiE;  MsHlon  CflfAfl 

Software  Engineering  Institute  CMMI^  DEV  vl.l/vl.2  -  SCAMPI  vl.WO  Class  A  Appraisal  Results  — ^ 

Maturity  Profile  by  Organization  Size 

Based  on  the  total  number  of  employees  within  the  area  of  the  organization  that  was  appraised 


100% 


Process  Maturity  Profile,  CMMI  for  Development  SCAMPI  Class  A  Appraisal  Results 
http://www.sei.cmu.edu/cmmi/casestudies/profiles/pdfs/upload/2010SepCMMI.pdf 


|  25  or  fewpr  ho  hi  to  7fi  Tib  m  ion  ini  m  ?nn  I  mi  to  m  mm  m  am  sni  to  innn  inni  to  2000  2000+ 

■  Not  Given  1  Initial  ■  Managed  □  Defined  |d  Quantitatively  Managed  a  Optimizing 

ion  4863  organizations  reporting  size  data 
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Are  Small  U.S.  Businesses  Using  CMMI? 


STRENGTH  T1IK01GII 1NDISTRY  &  TECHNOLOGY 


US  Appraisals 

Distribution  by  Org.  Size 

400  ■ 

■  Commercial 

■  Contractor  for  Mil/Gov 

■  Mil/Gov 

■  All  US  Appraisals 

_ iijIjIJIjI  II  lU  1 

\C 

/  «/  c/  </  </  </  </  ■/  y 

<<$  <&  #  A  ^  ^  v  * 

<o'  V  S?  \' 

U.S.  organizations  of  all  sizes  are  conducting  SCAMPI-A 
appraisals,  including  very  small  organizations. 

Large  %  of  U.S.  appraisals  conducted  in  all  domains  are 
by  small  organizations  (<100  people) :  commercial 
(59.5%),  govt,  contractors  (42.3%),  govt  (65.6%) 

CMMI  use  overseas  is  overwhelmingly  commercial. 
Within  U.S.,  predominantly  govt  contractors  (60.6%)  but 
with  significant  usage  in  commercial  industry  (30.9%). 


Based  on  data  from  published  SCAMPI  appraisal  results  for 
organjzations  -  which  could  be  part  of  a  large  company. 


US 

2,500-  1,500- 

10,000+  5,000-9,999  4,999  2,499 

1,000- 

1,499 

750-999 

500-749 

250-499  100-249  21-99  1-20 

Commercial 

0.3% 

0.3%  1.7%  2.0% 

1.7% 

1.7% 

3.1%  8.0%  21.1% 

41.0% 

18.5% 

Contractor  for  Mil/Gov 

0.6% 

2.5%  3.3%  3.5% 

4.6% 

2.9% 

6.1%  11.0%  22.6% 

30.1% 

12.2% 

Mil/Gov 

0.0% 

0.0%  1.0%  1.0% 

0.0% 

1.0% 

2.1%  11.5%  14.6% 

44.8% 

20.8% 

us 

10,000+ 

5,000-9,999 

2,500- 

4,999 

1,500- 

2,499 

1,000- 

1,499 

750-999 

500-749 

250-499 

100-249 

21-99 

1-20 

No  data 

Total 

Commercial 

1 

1 

6 

7 

6 

6 

11 

28 

74 

144 

65 

2 

351 

31% 

Contractor  for  Mil/Gov 

4 

17 

23 

24 

32 

20 

42 

76 

156 

208 

84 

5 

691 

61% 

Mil/Gov 

0 

0 

1 

1 

0 

1 

2 

11 

14 

43 

20 

3 

96 

8% 

All  US  Appraisals 

5 

18 

30 

32 

38 

27 

55 

115 

244 

395 

169 

10 

1138 

% 

0.4% 

1.6% 

2.6% 

2.8% 

3.3% 

2.4% 

4.8% 

10.1% 

21.4% 

34.7% 

14.9% 

0.9% 

Non-US 

U.S. 

Grand  Total 

Commercial/In-house 

3352 

352 

3704 

Contractor  for  Military/Government 

206 

690 

896 

Military/Government  Agency 

73 

96 

169 

Grand  Total 

3631 

1138 

4769 

Non-US 

U.S. 

Grand  Total 

Commercial/ln-house 

92.3% 

30.9% 

77.7% 

Contractor  for  Military/Government 

5.7% 

60.6% 

18.8% 

Military/Government  Agency 

2.0% 

8.4% 

3.5% 

Grand  Total 

100.0% 

100.0% 

100.0% 
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Are  small  businesses  using  CMMI  in  the  U.S 
defense  contracting  market? 


STRENGTH  THKOIGH 1NDISTRV  &  TECHNOLOGY 


US 

10,000+ 

5,000-9,999 

2,500- 

4,999 

1,500- 

2,499 

1,000- 

1,499 

750-999 

500-749 

250-499 

100-249 

21-99 

1-20 

No  data 

Total 

Commercial 

1 

1 

6 

7 

6 

6 

11 

28 

74 

144 

65 

2 

351 

31% 

IContractorfor  Mil/Gov 

4 

17 

23 

24 

32 

20 

42 

76 

156 

208 

84 

5 

691 

61% 

Mil/Gov 

0 

0 

1 

1 

0 

1 

2 

11 

14 

43 

20 

3 

96 

8% 

All  US  Appraisals 

5 

18 

30 

32 

38 

27 

55 

115 

244 

395 

169 

10 

1138 

% 

0.4% 

1.6% 

2.6% 

2.8% 

3.3% 

2.4% 

4.8% 

10.1% 

21.4% 

34.7% 

14.9% 

0.9% 

Data  from  SCAMPI  appraisal  results  is  for 
organizational  units  (Oils)  -  which  could  be  part  of 
a  larger  company.  Overall  company  size  is  not 
available  from  SCAMPI  results  filed  with  SEI. 

From  the  data  set  of  691  appraisals  by  U.S. 
government  contractors: 

•  428  were  by  OUs  <  200  people 

•  Known  large  companies  were  removed  from  this 
data  set 

•  *Best  judgment  is  265  appraisals  (62%)  were 
conducted  by  truly  small  companies  <  200  people 

•  This  265  appraisals  reflects  38%  of  the  complete 
data  set  (all  sizes,  691  appraisals) 

Even  though  these  reflect  estimates,  they 
substantiate  significant  usage  of  CMMI  by  small 
companies  in  the  U.S.  defense  market. 


Size 

(Employees) 

Total 

Appraisals 

Large 

Companies 

Truly 

Small* 

%  Truly 
Small 

<200 

428 

163 

265 

62% 

<  100 

296 

81 

215 

73% 

<50 

191 

42 

149 

78% 

<25 

111 

19 

92 

83% 

450 

400 

350 

300 

250 

200 

150 

100 

50 

0 


265 


163 


215 


81 


149 


42 


<200  <100 


<50 


92 


<25 


■  Truly  Small* 

Large  Companies 
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What  countries  conduct  the  most  small  appraisals? 

Number  of  appraisals  reported  with  organization  size  <100 


STRENGTH  TIIKOIGH  1NDISTRV  &  TECHNOLOGY 


China 

United  States 
India 
Spain 
Brazil 
J  apan 

Korea,  Republic  Of 

Taiwan 

France 

Mexico 

Argentina 

Malaysia 

Egypt 

Thailand 

United  Kingdom 

Colombia 

Chile 

Italy 

Canada 

Pakistan 

Germany 

Turkey 

Hong  Kong 

Sri  Lanka 

Viet  Nam 

Philippines 

Australia 

Singapore 


1161 

783 

313 

138 

132 

115 

102 

100 

93 

86 

69 

60 

36 

36 

34 

34 

33 

27 

25 

24 

20 

18 

14 

14 

13 

12 

11 

10 


70 

■  Malaysia 

60 

■  Argentina 

50 

■  France 

■  Korea,  Republic  Of 

40 

■  Japan 

■  Mexico 

30 

■  Brazil 

■  India 

20 

United  States 

10 

China 

0 


Small  US  Appraisals  (Orgs  <  SO) 

Distribution  by  Ore.  Size 


■  Commercial 

■  COniracior  lor 

■  MiI/Opv 
R  AH  US  Appro i sql* 


klJililllilllJ  I 
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Other  Observations 


Under-representation  of  small 
organizations  (CMMI  data) 
relative  to  small  US  businesses 
overall  (Census  data)  may 
represent  a  marketing 
opportunity 


There  is  more  proportional 
interest  among  small  commercial 
orgs.  than  other  orgs. 

(p  =  0.000  on  a  %2  test) 
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Distribution  by  Org.  Size 

CMMI  Appraisals  vs.  All  US  Businesses 


Org.  Size  Distribution  vs.  Sector 

100.00% 

80.00% 

60.00% 

40.00% 

20.00% 

0.00% 

>  200  1-200 


i  Commercial 
i  Contractor  for  Mil/Gov 
i  Mil/Gov 
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2010  CMMI®  Success  Factor  Survey 


•  A  Mixed  Method  Analysis  to 
Refine  an  Organizational 
Change  Model  for  Technology 
Organizations,  Doctoral 
Dissertation,  University  of 
Phoenix,  Dr.  Leigh  Riley,  2010 

•  Survey  and  analysis  of 
organizational  change  factors 
ensuring  successful 
implementation  of  CMMI® -Dev 
by  IT  organizations 

•  Further  analyses  will  be 
published  next  year  as  an 
addendum  to  the  Protean 
Model® 


•14  Permission  granted  by  the  author  for  use  by  NDIA  CMMI 
Working  Group  at  CMMI  Technology  Conference  2010 


Participant  Demographics 


Employment  Level 

Survey  Participants 

Number 

Percentage 

Executive 

5 

1% 

Management 

25 

34% 

Senior  Technical 

7 

9% 

Technical 

1 

1% 

Process  Improvement 

33 

45% 

Other 

3 

4% 

Total 

75 

100% 

Organization  Size 


A  Mixed  Method  Analysis  to  Refine  An  Organizational  Change  Model 

for  Technology  Organizations 

©2010  by  Leigh  Riley  ALL  RIGHTS  RESERVED 


Ranking  of  Organizational  Change  Factors  by  Size  of  the  Organization 


Rank 

Small 

Medium 

Large 

1 

Senior  Leadership  Support 

Senior  Leadership  Support 

Communication 

2 

Operational  Leadership 

Communication 

Senior  Leadership  Support 

3 

Communication 

Operational  Leadership 

Measuring  of  Progress 

4 

Training  the  Leadership 

Training  the  Employees 

Training  the  Employees 

5 

Training  the  Employees 

Measuring  of  Progress 

[Aligning  the  Initiative  with  Strategic  Goals 

6 

Measuring  of  Progress 

|  Aligning  the  Initiative  with  Strategic  Goals 

| Articulating  the  Benefits 

7 

Managing  Initiative  as  a  Project 

1  n  crem  entail  m  p  r  o  v  em  en  ts 

Managing  Initiative  as  a  Project 

8 

Aligning  the  Initiative  with  Strategic  Goals 

P  ro  v  i  d  i  n  g  S  u  p  p  o  it  M  ech  a  n  i  sm  s 

Operational  Leadership 

9 

Providing  Support  Mechanisms 

Managing  Initiative  as  a  Project 

P  ro  v  i  d  i  n  g  S  u  p  p  o  rt  M  ech  a  n  i  sm  s 

10 

Articulating  the  Benefits 

Training  the  Leadership 

Training  the  Leadership 

11 

1  n  crem  en  ta  1 1  m  p  r  o  v  em  en  ts 

Articulating  the  Benefits 

1  n  crem  en  ta  1 1  m  p  ro  v  em  en  ts 

12 

Involving  Employees 

Involving  Employees 

Involving  Employees 

13 

Changing  the  Performance  Management  System 

Changing  the  Performance  Management 

System 

Changing  the  Performance  Management 
System 

Factors  for  effective  organizational  change  are  similar  across  different  sizes  of 
businesses,  but  initiatives  may  have  shifting  priorities  (e.g.,  tactical  to  strategic) 


Permission  granted  by  the  author  for  use  by  NDIA  CMMI  A  Mixed  Method  Analysis  to  Refine  An  Organizational  Change  Model 

Working  Group  at  CMMI  Technology  Conference  2010  for  Technology  Organizations 

©2010  by  Leigh  Riley  ALL  RIGHTS  RESERVED 


Improving  Processes  in  Small  Settings 
(IPSS) _ 


STRENGTH  THKOK.il 1NDLSTRV  &  TECILNOLOGV 


•  “Proceedings  of  the  1st  International 
Researcher’s  Workshop  on  Process 
Improvement  in  Small  Settings” 

http://www.sei.cmu.edu/publications/documents/06.report 

s/06sr001.html 

•  “Improving  Processes  in  Small  Settings 
(IPSS):  A  White  Paper” 

International  Process  Research  Consortium  (IPRC), 
Software  Engineering  Institute 

>  Phase  1 :  Qualitative  study 

>  Phase  2:  Quantitative  research 

Barriers  to  entry: 

>  Lacking  core  competencies,  basic  practices 

>  PI  costs  within  operating  budget 


Reference:  Applicability  of  CMMI  to  Small  and  Medium  Enterprises 
Dr.  Rick  Hefner  and  Maggie  Glover.  SC  SPIN,  April  2010. 
http://www.uces.csulb.edu/SPIN/media/ppslide/Rick%2007CMMI%20Small%20Settinqs%20Part%20l.ppt 


>  Clear  roles/responsibilities,  flat  hierarchy 

>  Infrastructure 

>  Staff  culture 


Phase- 1  Results 


CMMI  SME  implementation  issues: 

■  Size  -  CMMI  is  considered  too  large  by  SMEs 

■  Linkage  to  SME  problems  is  not  immediately  evident 

■  Lack  of  SME  implementation  knowledge,  infrastructure  and  resources 
to  translate  process  framework  into  value  added  operational 
processes 

CMMI  Alternative  Approaches: 

■  CMMI-SME  Front-end:  Provide  direct  link  of  SME  development 
problems  to  the  appropriate  practices  in  the  model 

■  CMMI-SME  Back-end:  Link  of  model  components  to  off  the  shelf 
“whole”  product  solutions  appropriate  for  SME  implementation 

The  optimum  packaging  for  the  CMMI-SME  would  provide  “line  of 
si§ht”  connectivity  from  SME  problem  to  model  components  to 
implementation  solutions 


# 


Caniw^HsiVldfcitt 

Sftttwur*  Engineering  frwtltutti 


Pain  Associated  with  I  mproving  Existing 
Products  or  Developing  New  Products 


Average 

Response 

Standard 

Deviation 

Above  Average  "Pain" 

The  time  it  takes  to  get  to  market  /  recognize  revenue 

7.2 

2.0 

Trying  to  accelerate  corporate  growth 

6.7 

2.2 

Average  "Pain" 

Development  of  new  products  or  features 

6.0 

1.9 

Capitalization  /  funding  new  product  efforts 

6.0 

2.8 

Compliance  issues  w  ith  partners 

5.6 

1.9 

Quality  assurance  /  quality  control  issues 

5.4 

2.1 

Incorporating  increasing  systems  or  software  into 
new  or  improved  products 

5.3 

2.4 

Historical  New  Product  Failures 

5.2 

2.2 

Below  Average  "Pain" 

ISO  or  QS  9000  Compliance 

4.6 

2.3 

Canw^Melhnii 

sottwur*  Engineering  hslituta 


CMMI®  for  Small  Business 
NDIA  CMMI  Working  Group 


16 


Supporting  Small  Suppliers 


STRENGTH  THROUGH 1NDLSTRV  &  TK'HNOLOGV 


Acquirers  (such  as  prime  contractors)  often  use  CMMI  practices  to 
select,  manage,  and  support  their  small  suppliers. 


Typical  uses  of  CMMI  within 
CMMI  WG  member  companies: 

•  CMMI  maturity  level  requirements 
rarely  specified  to  subs  unless 
levied  by  external  customer 

•  Use  CMMI  practices  in  key  areas 
to  evaluate  supplier  risks 

•  Checklists  or  tools  used  for 
supplier  selection 

•  May  require  suppliers  to  follow 
prime  contractor’s  processes 

•  Provide  direct  support  to  suppliers 
-  training,  mentoring,  tools,  etc. 


Alignment  of  Processes  with  Subs 


Requirements  Management 

Project  Planning 

Project  Monitoring  &  Control 

Supplier  Agreement  Management 

Measurement  &  Analysis 

Product  &  Process  Quality  Assurance 

Configuration  Management 


CMMI  Process  Areas 
Level  3  (continued) _ 


organizational  rrocess  uetinition 
Organizational  T raining 
Integrated  Project  Management  for  IPPD 
Risk  Management 
Integrated  Teaming 
Integrated  Supplier  Management 
Decision  Analysis  &  Resolution 
Organizational  Environment  for 
Integration _ 


(4) 

(1) 

(1) 

(1) 

(4) 

(1) 

(4) 


Requirements  Development 
Technical  Solution 
Product  Integration 
Verification 
Validation 

Organizational  Process  Focus 

Notes: 


Organizational  Process  Performance 
Quantitative  Project  Management 


Organizational  Innovation  &  Deployment 
Causal  Analysis  &  Resolution  _ _ 


(5) 

(5) 


(1)  Subcontractor  internal  processes  and  IPT  operations  integrate  with  prime’s  processes. 

(2)  System  requirements  are  allocated  by  prime;  subcontractors  develop  requirements  at  the  configuration  item  (Cl)  level. 

(3)  All  subcontractors  integrate,  verify  and  validate  their  products  to  the  Cl  or  subsystem  level;  this  includes  integration  of  software  CIs  into 
hardware  CIs  or  line  replaceable  units  (LRUs).  Prime  integrates,  verifies  and  validates  at  the  system  level 

(4)  Subcontractors  follow  their  own  CMMI-compliant  business  processes. 

(5)  Prime  is  responsible  team’s  process  control  and  optimization. 


ISER-MLB-PR-07-143 


Reference:  “High  Maturity  System/Software  Cost  Estimation”, 
Richard  L.  W.  Welch,  PhD.,  Northrop  Grumman  Integrated  Systems 
CMMI  Technology  Conference,  November  2007. 
http://www.dtic.mil/ndia/2007cmmi/Wednesdav/4pmWelch.pdf 
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Resources  - 

Publications,  briefings,  social  networks 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Many  small  businesses  and  their  consultants  share 
their  CMMI  experience  in  publications  and  conference 
briefings. 

Over  40,000  Google  hits  on  “CMMI”  “small  business” 
Other  good  resources: 

•  SEI  -tech  reports,  guidance,  adoption  info,  case  studies,  ... 
http://www.sei.cmu.edu/cmmi 

•  NDIA  CMMI  Conference  briefings 

http://www.dtic.mil/ndia/ 

•  SEI  SEPG  Conference  briefings 

http://www.sei.cmu.edu/sepq/ 

•  CrossTalk  articles  http://stsc.hill.af.mil/crosstalk/ 

•  SPIN  meetings 

•  Newsgroups,  social  networks  (Yahoo!,  Linked  In,  Facebook,  ...) 

•  RSS  feeds 

•  Webinars  -  “Process  Improvement  at  the  Edges” 
http://www.sei.cmu.edu/librarv/abstracts/webinars/18dec2008.cfm 

There  is  no  shortage  of  good  people  willing  to  help  -  just  ask! 


A  Field  Guide  for  Improving 
Processes  in  Small  Settings 

Software  Engineering  Institute 
Carnegie  Mellon  University 
Pittsburgh,  PA  15213 

SuZ  Garcia,  Software  Engineering  Institute 

SEPG  North  America  2008 

Software  Engineering  Institute  CarnegieMcllon 


http://seispin.wikispaces.  com/file/vie 


Applicability  of 
CMMI  for  Small  to 
Medium  Enterprises 

SC  SPIN 
April  9,  2010 

Maggie  Glover,  Suz  Garcia  &  Gene  Miluk, 
Ph.D 

Software  Engineering  Institute 

Rick  Hefner,  Ph.G. 

Northrop  Grumman  Corporation 


http://www.uces.csulb.edu/SPIN/me 

dia/ppslide/Rick%2007CMMI%20S 

mall%20Settinqs%20Part%20l.ppt 
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Resources  - 

Guidance  for  Effective  CMMI  Adoption _ B3fjg.f.B3 


SEI  Resources: 

•  CMMI  in  Small  Settings  Toolkit  Repository 

http://seir.sei.cmu.edu/toolkit/index.html 

•  Reports  &  Technical  Notes  -  Interpretative  Guidance,  ... 

http://www.sei.cmu.edu/cmmi/casestudies/reports/ 

•  CMMI  Roadmaps  -  goal-driven  process  area  guidance 

http://www.sei.cmu.edu/librarv/abstracts/reports/Q8tnQ10.cfm 

•  Using  the  Software  CMM  in  Small  Organizations 

-  Mark  Paulk  papers:  SW-CMM,  but  still  good  guidance  (if  you  can  find  it) 

NDIA  Resources: 

•  The  Effective  Use  of  CMMI  -  NDIA  position  paper 

http://www.ndia.orq/Divisions/Divisions/SystemsEnqineerinq/Paqes/CMMI  Worki 

nq  Group. aspx 

•  The  Economics  of  CMMI  -  CMMI  WG  white  paper 

http://www.ndia.org/Divisions/Divisions/SvstemsEnqineerinq/Documents/Committ 

ees/CMMI%20Workinq%20Group/The  Economics  of  CMMI.pdf 

Consultant  Resources  ...  and  many  others 


CMMI®  for  Small  Business 
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Summary  and  Recommendations 


STRENGTH  T1IKOIGII INDLSTRV  &  TECHNOLOGY 


Perceptions  that  CMMI  is  too  burdensome  for  small  businesses  is  not 
supported  by  data  on  CMMI  adoption 

•  Greatest  growth  areas:  small,  commercial,  overseas 

•  No  evidence  that  CMMI  is  a  barrier  to  competition 

Small  businesses  face  many  of  the  same  obstacles  as  large  companies 

•  Sponsorship,  commitment,  resources,  alignment  with  business  objectives,  ... 

•  Tailored  processes  and  priorities  adapted  to  meet  the  needs  of  the  business 

•  Small  businesses  often  have  fewer  resources  to  address  these  challenges 

•  But  they  also  have  advantages:  agility,  innovation,  simplicity,  institutionalization 

Follow  proven  guidance  for  using  CMMI  effectively 

•  Focus  on  business  value  -  The  Economics  of  CMMI® 

•  Set  performance  goals,  measure  progress  against  them,  use  CMMI  to  improve 

•  Learn  from  others  -  take  advantage  of  the  many  resources  available  to  help 


CMMI®  for  Small  Business 
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For  More  Information 


STRENGTH  T1IKOIGII INDUSTRY  &  TECHNOLOGY 


NDIA  CM  Ml  Working  Group 

http://www.ndia.org/Divisions/Divisions/SvstemsEngineering/Pages/CMMI  Working  Group.aspx 


Jim  Armstrong 

Stevens  Institute 

Geoff  Draper 

Harris  Corporation 

Wendell  Mullison 

General  Dynamics, 
Land  Systems 


Steve  Austin 

Lockheed  Martin 

Jeffrey  L.  Dutton 

Jacobs  Technology 

Rick  Welch 

Northrop  Grumman 


Dan  Blazer 

SAIC 

Nancy  Fleischer 

Raytheon  Company 


Michael  Campo 

Raytheon  Company 

Kathy  Smith 

Hewlett  Packard  EDS 


With  special  appreciation  to  key  contributors  and  subject  matter  experts: 

Leigh  Riley,  Acme  Process  Group  Hal  Wilson,  Northrop  Grumman 

Mike  Phillips,  Software  Engineering  Institute  Rick  Hefner,  Northrop  Grumman 
Gene  Miluk,  Software  Engineering  Institute  Lynn  Penn,  Lockheed  Martin 

Bill  Van  Dalsem,  NASA  CMMI  Steering  Group 
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The  Tailoring  Dilemma 


Alan  Gellis 

alan.h.gellis@boeing.com 

Defense,  Space  &  Security 
Boeing  Military  Aircraft 
Mobility  -  Philadelphia 
November  17,  2010 


BOEING  is  a  trademark  of  Boeing  Management  Company. 
Copyright© 2010  Boeing.  All  rights  reserved. 


The  Tailoring  Dilemma 


■CMMI  tries  to  balance  flexibility  with  consistency  in 
adapting  an  organization’s  set  of  standard  practices 
to  a  project.  Decisions  are  at  the  project  level  per 
guidelines. 

■The  typical  corporate  approach  stresses 
commonality  and  high  level  approvals  of  changes. 

■The  Dilemma:  How  to  balance  commonality  and 
corporate  control  with  project  flexibility  and  low-level 
decisions 

■  This  is  a  “Money”  process  - 

■Do  it  right,  and  you  maximize  value 

■Do  it  wrong,  and  you  have  waste  and  inefficiency 


Copyright© 2010  Boeing.  All  rights  reserved. 
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The  Tailoring  Dilemma 


■The  CMMI  approach  to  Tailoring  is  that  a  standard  set  of 
processes  must  be  tailored  (adjusted  by  additions,  elaborations, 
deletions,  or  changes)  to  fit  the  needs  of  a  project 

■The  Corporate  approach  for  most  large  companies  (and  many 
others)  is  to  have  a  standard  set  of  processes  that  cannot  be 
changed  or  deleted  except  via  a  rigorous  approval  and 
documentation  process 

■CMMI  favors  management  of  processes  close  to  where  the 
work  is  done 

■The  Corporate  approach  favors  moving  more  of  the  process 
management  to  a  high  level,  central  area 

■These  differences  need  to  be  reconciled  for  both  sets  of 
objectives  to  be  met 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Recognition  of  Certain  Factors 


■  Solving  the  dilemma  requires  recognition  of  a  number  of  factors, 
and  then  developing  solutions  that  satisfy  (or  at  least  consider) 
each 

■  Not  all  processes  do  the  same  kind  of  thing 

-  Some  tell  how  to  do  work,  such  as  the  typical  engineering  process 

-  Some  control  business  requirements,  such  as  use  of  specific 
clauses  in  subcontracts 

-  Different  departments/  functions  may  have  different  needs 

■  May  require  different  treatment  of  additions/  elaborations, 
changes,  and  deletions,  and  different  disciplines 

■  Even  with  good,  approved  tailoring  guidelines,  may  need  to 
divide  tailoring  decisions  between  Project  Process  Group  and 
corporate  management 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Authority  based  on  trust  and  guidelines 


■  CMMI  provides  for  tailoring  based  on  guidelines 
approved  by  management  steering  group 

■  Corporations  delegate  authority  in  constrained  ways, 
based  in  large  part  on  trusting  those  given  the  authority 

■  Experience  and  track  record  of  individuals 

■  Processes  to  control  individuals/  incentives  to  follow 
direction 

■  Tiers  of  authority,  size  of  projects,  dollar  value,  risk 

■  Number  of  processes  in  an  area /  number  of  tailoring 
decisions 

■  Domain  &  department  differences  -  e.g.,  software  vs. 
hardware,  engineering  vs.  finance 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Project  Variation  a  factor 


■  Commercial  vs.  Defense  and  Space  projects 

■  Commercial  often  to  internal  standards,  so  less  need  to 
recognize  project  differences 

■  Defense  and  Space  often  have  more  project  and  customer 
variation,  more  need  to  tailor  to  recognize  differences 

-  Different  products  have  different  risk  levels  (manned  space 
vs.  legacy  aircraft  designs  or  ground  equipment) 

-  Spiral  vs.  waterfall  development 

-  “Clean  sheet  of  paper”  vs.  legacy  projects 

-  Team  stability,  experience  together  can  vary 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Cultural  and  Individual  Differences 


■  Many  companies  now  comprised  of  parts  with  different 
backgrounds 

■  Some  have  culture  of  micro-managing,  little  trust 

■  Some  delegate  considerable  authority,  great  trust 

■  Some  like  process  detail,  some  do  not 

■  Some  are  highly  disciplined,  some  are  not 

■  Some  have  lots  of  work;  some  do  not,  and  protect  their 
“rice  bowls” 

■  Some  have  experienced  workforce,  some  have  beginners 

■  Executives  approving  corporate  approach  to  tailoring 
are  influenced  by  their  own  backgrounds,  in  addition  to 
those  of  their  corporation’s  component  parts 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Tailoring  Documentation  Differences 


■  Typical  Process  Architecture  -  Before  Tailoring 


Organization/  Corporation 


Project/  Program 


A 


y  Organization’s 


J 


A 


y  Unique 


J 
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Tailoring  Documentation  Differences 


■  Typical  Process  Architecture  -  After  Tailoring  -  #1 


Organization/  Corporation 


Project/  Program 


A 


J 


Instantiation  of 
Organization’s, 
Including  Tailoring 


A 


y  Unique 


J 
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Tailoring  Documentation  Differences 


■  Typical  Process  Architecture  -  After  Tailoring  -  #2 


Organization/  Corporation 


Project/  Program 


A 


J 


Organization’s 


Unique, 

Including 

Tailoring 

Documentation 
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Tailoring  Documentation  Differences 


■  Typical  Process  Architecture  -  After  Tailoring  -  #3 


Organization/  Corporation 


Project/  Program 

-  ^ 

— — v  Organization’s 

_  J 


Tailoring  in 
Separate  Document, 
Such  as  Plan  or  Web  Page 


A 


y  Unique 


J 
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Resolution  Process 


■  Two  parts: 

■  The  project’s  tailoring  guidelines 

■  The  corporation’s  and  division’s  (the  organization’s) 
standard  processes 

■  Authority  to  tailor  flows  down  from  the  corporate 
management  through  tiers  of  its  organization  to  a 
Process  Group  governing  a  project 

■  Tailoring  authority  must  meet  diverse  interests  of  the 
corporation 

■  Must  also  meet  the  needs  of  the  project  for  it  to  be  CMMI 
“compliant”  and  maximize  value  to  the  project 


Copyright© 2010  Boeing.  All  rights  reserved. 
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Resolution  Process  (continued) 


■  Major  consideration  is  how  good  and  how  disciplined 
the  corporation’s  standard  processes  are  to  begin  with. 

■  If  the  corporation  has  good  process  discipline  and  many 
practitioners  trained  in  sophisticated  techniques,  it  is 
easier  to  provide  clear  rules  for  tailoring  because  many  of 
the  issues  have  already  been  solved 

■  A  corporation  with  rambling,  confusing  process  and 
procedure  documents  that  just  grew  over  time  will  have 
difficulty  fitting  tailoring  provisions  into  them 

-  Adding  tailoring  words  to  confusing  text  will  still  look 
confusing  and  be  hard  to  understand  and  follow 

-  When  you  consider  tailoring,  it  often  highlights  the  need  to 
overhaul  process  and  procedure  documents 

■  Most  corporations  are  probably  somewhere  in  middle 
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Special  Factors  in  Resolution 


■  Individual  executive  personalities  are  real  (e.g.,  she  has  a  “won’t  of  iron”  -  she  won’t  give  any 
flexibility) 

■  Standard  process  SMEs  often  have  views  (I  would  not  have  written  that  if  I  didn’t  want  everyone 
to  do  it  exactly  that  way.) 

■  Cost  is  a  major  consideration,  but  not  always  evaluated  in  same  way 

■  CMMI  is  often  viewed  as  expensive,  so  authority  should  consider  cost  containment 

■  Authority  to  add  or  elaborate  may  result  in  unnecessary  added  costs 

■  Frequent  process  improvement  activity  can  be  “churning” 

■  Process  changes  can  result  in  need  for  more  training  expenditures 

■  Appraisal  and  audit  costs,  including  artifact  costs,  can  rise  with  more  tailoring 

■  May  result  in  unnecessary  process  differences  across  project  or  among  similar  projects,  resulting 
in  added  costs 

■  More  tailoring  increases  process  documentation  costs 

■  Corporations  ask:  Which  process  expenditures  produce  the  most  value?  CMMI  is  not  the  only 
choice. 

■  Is  a  business  case  needed  for  justification? 

■  Does  authority  to  delete  or  change  contribute  to  added  risk?  Is  any  added  risk  properly 
managed? 

■  How  does  CMMI  activity  fit  with  non-CMMI  activity? 

■  What  is  relationship  to  computer  tools  and  their  cost? 

■  Is  project  workforce  mostly  experienced  or  “beginners”? 

■  Is  customer  involvement  present? 

■  Does  culture  favor  detail  or  brevity? 
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How  to  proceed 


■  Need  to  identify  the  organization’s  set  of  standard  processes  and  evaluate  their 
suitability  first 

■  If  they  are  inadequate,  we  are  talking  about  process  changes  and  process 
improvements  to  the  standard  set  before  we  ever  get  to  tailoring  for  a  project 

■  Need  to  put  specific  tailoring  rules  into  organization’s  set  of  standard  processes  in 
order  to  enable  them  to  be  tailored  in  a  predictable,  disciplined  way. 

■  Top  level  for  entire  corporation  can  take  a  long  time  to  coordinate,  and  can  become 
too  general 

■  Limiting  to  one  division  or  business  unit  may  not  have  the  right  authority  for  higher 
level  documents,  but  is  easier  to  get  approved 

■  Choice  of  which  standard  process  document  to  modify 

■  Choice  of  where  to  document  tailoring  -  relates  back  to  Organization’s  standard 
processes 

-  For  users  to  find  it 

-  For  information  for  other  projects 

-  For  configuration  control 

-  Relationship  to  other  documents  that  may  be  modified 

-  May  be  separate  issue  of  where  to  document  approval 

-  Relationship  to  process  improvements 

■  Need  to  get  attention  of  higher  management  and  get  on  their  priority  list 
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Result 


■  Tailored  processes  at  project  level  fit  into  spectrum  of 
higher  level  processes,  same  level  processes,  lower 
level  processes  and  work  instructions,  standards,  tool 
instructions,  training  materials,  library  or  knowledge 
management  assets,  etc. 

■  I  would  like  to  be  able  to  say  that  there  is  a  perfect  way 
to  solve  the  Tailoring  Dilemma,  but  there  isn’t. 

■  The  best  we  can  do  is  to  indentify  all  of  the  applicable 
requirements  and  constraints  and  work  through  them 
in  a  rational  and  consistent  way  that  meets  the 
requirements  and  provides  value. 

■  There  will  always  be  some  people  who  will  be  unhappy. 

■  Don’t  be  one  of  them! 
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Building  Example 


NORTHROP  GRUMMAN 


•  Joe  Process  wants  to  build  a  building:  rf F3 

•  How  many  stories  (levels)  in  this  building? 

•  How  many  levels  initially?  2  Levels  or  Stories 

•  Does  he  have  enough  information  to  build  the  maximum 
number  of  levels  (Say  5  levels  is  the  maximum)? 

•  What  is  his  implementation  plan  to  minimize  re-work  if  after  building 
initially,  Joe  chooses  to  add  additional  levels  to  the  building? 

•  What  are  the  plumbing  requirements  for  a  5  story  building? 

•  Should  all  of  these  requirements  be  considered  for  the  initial  2 
story  building? 
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Implementing  CMMI® 


NORTHROP  GRUMMAN 


•  How  do  YOU  implement  a  Process  System  using  the  CMMI  (and  other 
process  standards  and  requirements) 

-  Lots  of  folks  take  it  one  level  at  a  time 

•  Advantage  -  a  very  methodical  approach,  it  allows  you  to  focus  on 
immediate  process  needs,  helps  build  confidence  in  implementation 

•  Disadvantage  -  as  you  progress  to  higher  levels,  you  may  discover  that 
a  lower  level  implementation  needs  some  rework  or  may  not  be  robust 
enough  to  support  the  higher  level  implementation 

-  However,  a  systems  approach  would  suggest  that  decision  analysis  be  used 
to  select  the  best  implementation 

•  The  CMMI  practices  (requirements)  for  ALL  levels  are  known  in  advance 

•  Analyze  all  requirements,  and  develop  a  concurrent  (and  phased) 
approach  to  optimize  implementation  cost  by  minimizing  re-work 

-This  is  done  by  making  a  trade  off  of  re-work  vs.  upfront  costs 


®  CMMI  is  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
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Viewing  CMMI  Requirements  by  Levels 


NORTHROP  GRUMMAN 
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Viewing  CMMI  Requirements  by  Categories 


NORTHROP  GRUMMAN 
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Viewing  CMMI  Requirements  by  Functions 


NORTHROP  GRUMMAN 


Analyze  CMMI  Requirements  (Practices)  to  determine  basic  functions 


-  Measure 

-  Analyze 

-  Reporting 

-  Evaluation 

-  Develop 

-  Review 

-  Document 

-  Maintain 

-  Gain  Commitment 

-  Plan 


-  Report 

-  Select 

-  Establish  Teams 

-  Communicate 

-  Monitor 

-  Deploy 

-  Train 

-  Achieve  Traceability 

-  Control 

-  Etc 


Perform  Basic  Functional  Analysis  and  Allocation  of 

CMMI  Requirements 
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Viewing  CMMI  Requirements  by  Functions 


NORTHROP  GRUMMAN 


•  Analyze 

•  Gain  Commitment 

•  Achieve  Traceability 


•  Analyze 

•  Measure 

•  Develop 

•  Review 
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Functional  Analysis  and  Allocation 


NORTHROP  GRUMMAN 


Flan 

IPM  SP  1.4-1  Integrate  other 
plans  that  affect  the  project 
with  the  project  plan. 

OPF  SP  1 .2-4  Plan,  schedule, 
and  prepare  for  the  process 

OID  SP  1.3-1  Plan  the  pilots. 


:e 

IPM  SP  2.2-5  Analyze  the' 
special  cause  of  process 
variation  to  determine  the 
ms  the  anomaly  < 
REQK^PTi=yAfiaIyze 
requirements  to  ensure  that  th| 
ilisRed  criteria  aril 
)PD  SP  1 .4-1  Determine  th«h 
organization's  needs  for  storing 
retrieving,  and  analyzing 
leasurements. 


Gain  Commitment 

PMC  SP  2.2-3  Negotiate 
changes  to  internal  and 
tai  uummitmarrter 
REQM  SP  1.2-4  Negotiate 

OPF  SP  2.2-2  Negotiate  and 
document  commilmGiiiLs 
among  the  process  station 
teams  and  revise  their  pr 
action  plans  as  necessary.' 


MASP  1.4-4  Review  and 
update  the  proposed  content 
and  fo  rim  a  e  of  the  specified 
analyses  and  reports. 

/ER  SP  3.2-4  Record  all 
results  of  the  analysis  in  a 
report. 

CM  SP  1.2-6  Create 
configuration  management 
reports  from  the  configuration 
management  system. 


Measure 

CAR  SP  2.2-1  Measure  the 
change  in  the  performance  of 
the  project's  defined  process 
as  appropriate. 

DAR  SP  1 .4-3  Determine  the 
measures  needed  to  support 
aluation  motlic 
OPD  SP  1 .4-6  Enter  the 
specified  measures  into  the 
^  repository. 


Maintain 

PI  SP4^2-5  Maintain  the 
product  integration 
environmentthroughout  the 

TEQM  SP  1 .3-2  Maintain  the 
juirements  change  histor 
withtRe  rs 
changes. 

IPM  SP  1.3-3  Maintain  the 
qualification  of  the  components 
of  the  project’s  work 
environment. 


Monitor 

SAM  SP  2.1-6  Monitor  risks 
involving  the  supplier  and  take 
corrective  action  as  necessary. 
RSKM  SP  3.2-1  Monitor  risk 

)PIM  SP 2.3-2  MonttorchanE 
in  quality  and  process- 
performance  objectives  and 
selected  subprocess’  process 
capability. 


Document 

CM  SP  1 .3-3  Document  the  set 
of  configuration  items  that  are 
contained  in  a  baseline. 

DAR  SP  1 .2-6  Document  the 
rationale  for  the  selection  and 
rejection  of  evaluation  criteria. 
MASP  1 .1-1  Document 
information  needs  and 
objectives. 
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Examples  of  Functions 


NORTHROP  GRUMMAN 


Measure 

CAR  SP  2.2-1  Measure  the 
change  in  the  performance  of 
the  project's  defined  process  as 
appropriate. 

DAR  SP  1 .4-3  Determine  the 
measures  needed  to  support  the 
evaluation  method. 

OPDSP  1.4-6  Enter  the 
specified  measures  into  the 
repository. 


Analyze 

QPM  SP  2.2-5  Analyze  the 
special  cause  of  process 
variation  to  determine  the 
reasons  the  anomaly  occurred. 
REQM  SP  1.1-3  Analyze 
requirements  to  ensure  that  the 
established  criteria  are  met. 
OPD  SP  1 .4-1  Determine  the 
organization's  needs  for  storing, 
retrieving,  and  analyzing 
measurements. 
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Measurement  Function 


NORTHROP  GRUMMAN 


Establish  measures  and  objectives  for  determining  the  value  of  each 
process  and  technology  improvement  with  respect  to  the  organization’s 
quality  and  process-performance  objectives. 

Measure  the  value  of  each  process  and  technology  improvement. 

Measure  the  change  in  the  performance  of  the  project's  defined  process  as 
appropriate. 

Identify  how  process  performance  is  to  be  measured 

Identify  the  measures  that  are  appropriate  for  statistical  management. 

Revise  the  measures  and  statistical  analysis  techniques  as  necessary. 

Store  process  and  product  measures  in  the  organization’s  measurement 
repository. 

Define  a  common  set  of  process  and  product  measures  for  the 
organization's  set  of  standard  processes. 

Collect  performance  measures  on  the  risk-handling  activities. 

Update  measures  and  measurement  objectives  as  necessary. 

Obtain  the  data  for  base  measures. 

Review  the  results  of  collecting  and  analyzing  measures  for  controlling  the 
project. 


DID  SP  2.1-4 


01 D  SP  2.2-1 
CAR  SP  2.2-1 


OPD  SP  1 .4-2 

RSKM  SP  3.2-0 
MASP  2.1-1 
MASP  1.4-5 
PMC  SP  1 .6-2 
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Analyze  Function 


NORTHROP  GRUMMAN 


Analyze  the  costs  and  benefits  of  potential  innovative  improvements 

Analyze  selected  defects  and  other  problems  to  determine  their  root 
causes. 


OIDSP  1.2-4 


Establish  and  maintain  the  organization’s  process-performance  baselines 
from  the  collected  measurements  and  analyses. 


OPP  SP  1 .4-2 


Analyze  the  interaction  of  subprocesses  to  understand  the  relationships 
among  the  subprocesses  and  the  measured  attributes  of  the  subprocesses. 


QPM  SP  1.2=3 


Analyze  the  special  cause  of  process  variation  to  determine  the  reasons  the 
anomaly  occurred. 

Select  the  methods  based  on  the  purpose  for  analyzing  a  decision  and  on 
the  availability  of  the  information  used  to  support  the  method. 

Analyze  the  organization's  common  set  of  measures. 

Analyze  issues  to  determine  need  for  corrective  action. 

Analyze  the  impact  of  changes  and  fixes  proposed  in  the  change  requests. 

Specify  and  prioritize  the  analyses  that  will  be  conducted  and  the  reports 
that  will  be  prepared 

Analyze  requirements  to  ensure  that  the  established  criteria  are  met. 


QPM  SP  2.2-5 


REGM  SP  1.1-3 
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Analyze  Defects  Function 


NORTHROP  GRUMMAN 


Analyze  Defects  or  other  issues  for  preventive  action 


Analyze  Defects  or  other  issues  for  corrective  action 


In  Implementing  the  Analyze  Defect  function  we  can  now 
decide  whether  to  address  this  function  for  corrective  only  or 
for  both  corrective  and  preventive  action  with  full  knowledge 

of  all  the  requirements 
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Summary:  “Vertically  Slicing  the  CMMI®” 
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Since  all  CMMI  requirements  are  known  beforehand  the  following  holistic 
approach  can  be  employed  to  proactively  build  the  foundation  for  all  levels: 

•  Analyze  the  CMMI  practices  across  all  maturity  levels, 

•  Understand  their  interrelationships 

•  Decompose  the  CMMI  into  functions  with  allocated  requirements  otherwise 
known  as  Functional  Analysis  and  Allocation  (FAA) 

•  Plan  for  a  time  phased  optimal  implementation  by  making  decisions  about 
what  requirements  will  be  implemented  in  each  phase 

Therefore,  "Vertical  Slicing  the  CMMI"  really  equates  to  using  good  system 
engineering  principles  to  enable  good  decisions  on  what  requirements  are 
addressed  in  each  phase  of  implementation  so  that  re-work  in  minimized  as 
progressively  higher  CMMI  levels  are  implemented 
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Enterprise  Process  Integration 


Measurement  Planning  and  Data 

Collection 
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Topics 


*  Background 

*  Deploy  measurement  process  across 
programs 

*  Collect  measurement  usage  across  the 
company 

*  Evaluate  measures  for  commonality  and 
standardization 

*  Summary 


Enterprise  Process  Integration  > 

EPi 
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Lockheed  Martin  Aeronautics  Overview 
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Background 


•  Lockheed  Martin  Aeronautics  measurement  process 
improvements 

-  Multi  site  measurement  programs  -  pre  2000 

-  Programs  specific  measures  and  formats  -  pre  1993 

-  SEI  CMM  Level  3  for  software  -  1993 

*  SW  Standard  Metrics 

-  SEI  CMM  Level  4  for  software  -  1999 

*  SW  Quantitative  Management 

*  Common  Repository  for  all  SW  measurement 
data 

-  SEI  CMMI  Level  3  at  company  level  -  2007  &  2010 

*  Company  wide  measurement  process 
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Use  of  measures  across  the 
company 


•  How  to  collect  data  on  measurements  used 
across  the  company  and  provide  value  added  to 
the  programs 


-  Institutionalize  the  Measurement  &  Analysis 
(M&A)  process 

-  Identify  common  measurement  needs  across 
the  programs 

-  Standardize  M&A  planning  across  the  company 

-  Streamline  M&A  planning  for  the  programs 


Enterprise  Process  Integration  > 

EPi 
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Approach 


*  Establish  a  method  to  collect  data  on  measurement 
indicators  while  using  it  to  pre-populate  program 
M&A  plans 

-  Establish  Measurement  Model  Database 

*  Measurement  Objectives,  Information  Needs, 
and  Measurement  Indicators 

-  M&A  planning  workshops  on  site  with  the 
programs  facilitated  data  collection  and 
distribution  of  common  measurement  planning 
data 

-  Pre-populate  M&A  planning  templates  using  data 
in  the  database 


Generate  Measurement  Matrix  for  each  program 
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X 


Approach  (cont.) 

*  Measurement  Matrix  (MS  Excel  Work  Book) 

-  Measurement  Objectives 

-  Information  Needs 

-  Measurement  Indicators 

-  Base  Measures 

-  Program  briefings  where  indicators  are  used 

-  Program  owners  of  the  indicators 

*  Matrix  is  a  part  of  program  M&A  plans 

-  Replaces  multiple  traceability  tables 

*  Matrix  is  generated  from  Measurement  Model 
database 
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Approach  (cont.) 


Enterprise  Process  Integration 

EBi 


•  Resources: 

-  Pre-populated  M&A  plan  template 

*  Common  objectives,  information  needs  and 
measures 

*  Boiler  plate  information 

*  Training 

*  References  to  standard  process 

-  Measurement  Model  Database 

*  Specification  data  for  standard  measures 

*  Common  objectives  and  information  needs 

*  Able  to  generate  tables  for  M&A  plan 

*  Collect  program  specific  measures 
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Measurement  Plan  Template 


•Standard  Measures 
•Standard  tools 
•Documentation  standards 
•Organization 
•Process  standards 


Measurement  Plan 

Template 

(Pre-populated) 


Enterprise  Process  Integration  > 

EPJ 
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Measurement  Model 


Stores  information  about 
the  program 


X 


Stores  information  about 
the  objectives,  information 
needs,  indicators  and  base 


Tabular  reports  can  be 
generated  consistent  with 
those  needed  for  M&A 
Planning 


Can  be  used  to  generate  a  Measurement  Matrix  for  any  program  or  function 


Enterprise  Process  Integration  > 

EPi 
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Measurement  Model  (cont.) 


r\/i 


Collect  information  about 
the  objectives 


Measuremept-afidAnalysis  Model 


List  the  objectives 


X 


[ _ J  Objectives  Input 


| _ I  information  Needs  Input 


m  LM  Measurements  Library 


Proactively  Manage  Configuration 
Control  of  Each  Aircraft  and  the  RERP 
(C-5M)  Baseline 

Manage  timely  Procurement  of  all 
supplier  required  products  for  C-5 
programs 


Minimize  Waste  to 
Affordable  Product 

=1  i~l  P  i— i  h  nKi  i h ■=■  I- 1— i  P 


Collect  information  about  the 
Measurement  Indicators 

List  the  Measurement 
Indicators 


Enterprise  Process  Integration  ♦ 

EPJ 
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Measurement  Model  (cont.) 


Descriptions: 


Information  Needs: 


Is  the  program  performing  to  schedule? 


Relates  the  Objective  to 
appropriate  Information 
Needs 


Is  the  aircraft  meeting  quality  expectations? 


Does  the  corrective  action  process  work  effectively  to 
meet  program/project  expectations?  v 


Have  the  defects  been  corrected/resolved  appropriately 
for  the  project's  needs?  v 


How  much  scrap  &  rework  is  being  generated? 


Are  staffing  &  skills  meeting  program  needs? 

F  ro  gram/P ro j  e  c  r  o  b j  e  c  th  e  s/ris  ksA is  s  11  e  s 


Relates  the  Information 
Need  to  its  category, 
Indicators  used  to  answer 
it ,  and  the  Objectives  it  is 
related  to 


Enterprise  Process  Integration  > 

EPi 


Title  |ls  the  program  performing  to  schedule? 

Assigned  To  |  v 

Requested  By  v 

Category  ISchedule  and  Progress  v 


Add  Domain 
Add  Function 


Item  ID  1  fcH]  Type 

v  [  Add  Priority 


Double  Click  on  Category  to  view  Detail  Information 


v  J  Add  WorkProduct 
VUGS 


Info  Type:  [Program  Specific 


m  Indicators  Selection/ Review 


[X~]  Delete  Current  Information  Need 


View  Report  Format 


Description  |  Measurement  Concept  Analysis  Requirements  Action  Requirements  Comments 


Information  o 
completion. 


ing  tasks,  releases,  and  activities  according  to  scheduled  or  planned 


Objectives: 

m 

► 

deliver  High  Quality  Aircraft  on  Schedule 

Aggressively  Manage  GFE  and  DMS  Issues 

\ 

— 

H  |  Delete  Objective 

- 

Indicator: 

- 

CDRL  Status 

V 

CD 

Development  Progress 

V 

Software  Requirement  Progress 

V 

Software  Change  Request  Status 

V 

Flprtrnnir  "Inh  Hplav  Si  immarw 
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Measurement  Model  (cont.) 


x 


LA. 


LM-Aero  Library  of  Indicators  -  Detail  Information 


Close 

Form 


Measures 


Wiring  harness  completion  status 


Change  Status  of  current 
measure 


OSP: 

Table  References: 

MM 

Program  Specific 

V 

Measurement  Category: 

Process  Performance  v 

Appendix 

N/A 

Req  u  i  red  /O  ptio  na  1: 

0  v 

This  measures  Answers  Questions  Such  as: 

Are  we  completing  wiring  harneses  as  needed? 

Implementation  Guidance:  Analysis  Frequency: 

Storage  Requirements: 

Collection  Levet 

Collection  Requirements: 

CM  MI  Process  Areas: 


X  Delete  Current  Indicator 

]  Add  New  Indicator 
[&]  LM  Measurement  Library 

0  All  Indicators  Report 
0  Current  Indicator  Report 
rogram  Indicators  Rpt 
Function  Indicators  Rpt 


Indicator  Tool 


Collect  detailed  information  about  each  measurement  indicator 
Can  generate  specification  table  for  each  or  all  indicators 
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Measurement  Matrix 


HI 

Y 

Z  AA  AB  AC  AD  AE  AF 

1 

Measures 

ireORO 

\ppendi 

Categori 

Question 

Purpose 

Objective 

□escriptic 

2 

3 

4 

5 

6 

7 

8 

8 

10 

11 

12 

13 

14 

15 

16 

17 

18 

Automated  hiequest  tor  tngineenng 
Action  Metrics 

R 

A 

Process  Performance 

E.S.001 

AREA  Standard  Metrics  Deck  includes  15  me 

Track  AREAs  to 

Multiple  barHI 

BTP  Commit  Plan 

R 

A 

Process  Performance 

E.S.001 

Tracks  performance  to  the  Commit  Plan.  Thi; 

Track  BTP  perfrc 

Chart  showin 

BTPNeed  Plan 

R 

A 

Process  Performance 

E.S.001 

Tracks  performance  to  Meed  Plan.  The  latest 

Tracks  drawings  | 

Tracks  drawii 

Cash  us.  Time 

0 

A 

Resource  and  Cost 

B.S.008;  B.S.010 

Determine  whether  program  is  meeting  comp; 

Prouide  executive 

Plan  us.  ActL 

CM  CCB  CR  Cycle  Time 

R 

A 

Process  Performance 

E.S.004;  E.S.  005;  E.S. 

Track  the  performance  of  the  formal  change  r 

Identify  potential  i 

This  measurt 

CM  OOS  CR  Disposition  Index 

R 

A 

Process  Performance 

E.S.005;  E.S.  006 

Tracks  the  disposition  of  formal  CRs. 

Identify  potential  i 

Compares  th 

Cost  Performance 

C 

A 

Resource  and  Cost 

B.S.001;  B.S.002;  B.S.C 

Prouide  insight  into  the  actual  labor  expenditu 

Identify  risks  to  th 

The  earned  u 

Cost  Performance  Index 

C 

A 

Resource  and  Cost 

B.S.001;  B.S.002;  B.S.C 

Represent  actual  cost  expenditure  to  budgete 

Identify  risks  to  th 

CPI  indicate; 

Cost  Variance 

C 

A 

Resource  and  Cost 

B.S.001;  B.S.002,  B.S.C 

Identify  uariance  of  actual  labor  expenditures  t 

Indicate  if  project 

This  earned  v 

CPU  Throughput 

C 

A 

Product  Size  and  Stability 

C.S004;C.S.005;  C.SI 

Compare  actual  to  expected  computer  resoui 

Identify  computer 

This  measurt 

Defect  Burndown 

C 

A 

Product  Quality 

D.S.003;  D.S.006,  D.S.I 

Identify  how  long  it  will  take  to  correct  the  defe 

Identify  impacts  t< 

The  Defect  E 

Defect  Profile  Table 

R 

A 

Product  Quality 

D.S001;  D.S.002;  D.S.O 

Identify  how  many  defects  are  detected  and  inj 

Represent  produ< 

The  defect  pr 

Defects  by  Category 

R 

i“ 

A 

A 

Product  Quality 

D.S.001;  D.S.004 

1“ i  C  1“ i  C  fifiO 

Identify  the  number  of  defects  per  defect  cate< 

Prouide  insight  in 

Column  grap 

Deliuery  Status 

L- 

0 

H 

A 

Product  Quality 

Schedule  and  Progress 

U.o.UUd;  U.-i'.UU  J 

A.S.008;  A.S.009 

Identify  the  number  of  defects  per  defect  seue 

Indicates  whether  final  product  deliverables  ar 

Prouide  insight  in 

To  control  final  o 

The  Defects 

Count  of  plar 

Earnings  Before  Interest  &  Taxes  (EBIT) 
us.  Time 

0 

A 

Resource  and  Cost 

B.S.008;  B.S.008 

Determine  whether  program  is  meeting  comp; 

Prouide  executive 

Plan  us.  ActL 

InputfOutput  Utilization 

c 

A 

Product  Size  and  Stability 

C.S.004;  C.S.005;  C.S.I 

Compare  actual  to  expected  utilization  of  inte 

Identify  computer 

This  measurt 

isi—ti—t  itpif.  a  i-rni 

i—  nrn  i—  !”■  riri-"i  i—  r-  m 

h  i-i  i/r-ir-i  iTnF.  a  iti- ii  i- ■  a  t  _  _  i.  _  :  _  .  i  t-tt  - -  i— ^  nrn  i—  r-  r«  n  -i  i—  «-■  rirt  t _ i.  bi.  _ _ r _ _  _  -  -  i _ i _ : _ f.  ^ _ _ _ _  l.  is  ...  i — ■ _ r _ 

h  ^  h  Objective-Info  InfoNeeds  InfoMeasures  AOTMeasures  IndicatorToob  Indicators  a  se  BaseMeasures  Questions  [i| 


Generated  from  the 
Measurement  Model 
database 

Each  tab  replaces  a 
table  in  the  M&A  Plan 


•Measurement  Objectives  and  their  associated  Information  Needs 

•Information  Needs  mapped  to  standard  categories  and  the  Indicators  used  for 

each  Information  Need 

•Specification  details  about  each  Indicator 

•Base  Measures  used  for  each  Indicator 
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Relationship  of  Measurement  Matrix 

Tables 

InfoMeasures 


Objectives-lnfo 

Multiple 


Objectives 

Source 

(Company/ 

Program/ 

Customer) 

Information  Needs 

1 .  Perform  to  Customer  Expectation 

Customer 

a.  What  is  the  customer  expectation  of  program  ^ 
performance? 

b.  How  is  the  delivered  aircraft  performing‘s^ 

c.  Are  the  delivered  services  meeting 
expectations? 

2.  Manage  Scrap  &  Rework 

Company 

a.  How  much  scrap  and  rework  is  beina 

generated  on  the  program?  ^ 

3.  Perform  to  Schedule 

Program 

a.  Is  the  program  performing  to  schedule? 

4.  Perform  to  Budget  Expectations 

Program 

a.  Is  the  program  performing  to  budget? 

5.  Perform  to  Quality  Expectations 

a.  Do  the  work  products  have  adequate  quality? 

b.  Are  the  delivered  products  performing  to 
quality  expectations? 

expectation? 

X 


Once 

Multiple 

Information  Needs 

Measure 

a.  What  is  the  customer  expectation  of 
program  performance? 

CPAR  ' — 

Delivery  Status^^^v^. 

Field  Reports 

b.  How  is  the  delivered  aircraft 
performing? 

Field  Reports 

FOD  found  at 

c.  Are  the  delivered  services  meeting 
expectations? 

MTBR  N. 

Mission  Capability  Rate 

d.  How  much  scrap  and  rework  is  being 
generated  on  the  program? 

Scrap  and  Rework 

FOD  found 

Lost  Tools 

e.  Is  the  program  performing  to 
schedule? 

Schedule  Performance  Index 

Schedule  Health 

f.  Is  the  program  performing  to  budget? 

Cost  Performance  Index 

Cost  Variance 

Match 


AIIMeasures 

Once  Multiple 


Measure 

Information 

Category 

Base  Measures 

CPAR 

Customer  Expectations 

Rating  color  for  each  element  ratecL 

Delivery  Status 

Customer  Expectations 

#  of  aircraft  expected  each  month  ^ 

#  of  aircraft  delivered  each  month 

Field  Reports 

Customer  Expectations 

Defective  item  ^ 

Defect  severity 

Defect  category 

Date  found 

FOD  Found  at  delivery 

#  of  FOD  items  found  each  month 

MTBR 

Date  defect  found 

Date  repair  completed 

BaseMeasures 

Once  Once 


Base  Measures 

Measure 

OMR 

Collection  tool 

Rating  color  for  each 
element  rated 

CPAR 

N 

PPADB 

#  of  aircraft  expected 
each  month 

Delivery  Status 

N 

MS  Excel 

#  of  aircraft  delivered 
each  month 

Delivery  Status 

N 

MS  Excel 

Defective  item 

Field  Reports 

Y 

ALIS 

Defect  Severity 

Field  Reports 

Y 

ALIS 

Date  category 

Field  Reports 

Y 

ALIS 

Match 

'Match 

once  v  InformationNeeds 


Information  Need 

Category 

Measurable 

Concept 

Questions 

Rating  color  for  each 
element  rated 

Customer  Expectations 

Identify  customer 

specified 
criteria  on  key 
elements 

Are  we  meeting 
customer 
performance 
expectations? 

#  of  aircraft  expected 

Schedule  and  Progress 

Compare  #  of  aircraft 
delivered  to  # 
of  aircraft 
deliveries 
planned 

Are  we  delivering  the 
number  of 
aircraft 
planned? 

Defective  item 

Product  Quality 

How  many  defective 

been  found? 

Cost  Performance  Index 

Resources  and  Cost 

Compare  actual  cost 
expended  to 
planned  cost 
expended 

Are  actual  costs 
within 

tolerance  of 
expected  cost 
expenditures? 

Each  Objective  may  map  to  multiple  Information  Needs.  Each  Information  Need 
can  map  to  multiple  Indicators.  Each  Indicator  may  use  multiple  Base  Measures. 
Indicator  details  includes  an  owner  for  each  measure  and  what  management 
briefings  include  the  measure.  Indicators  can  be  sorted  by  owner,  briefing, 
r,  etc  for  program  use. 
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Program  Measurement  and  Analysis  P\an — r 


Measurement  Matrix  is  combined  with  Measurement  Plan 
template  to  create  program  Measurement  Plan 


Organizational  Analysis  of  Measurement 

Planning  Data 

Load  program  measurement  matrices  into  organizational  Measurement 
Model 


Organizational  Analysis  of  Measurement 

Planning  Data 


•Evaluate  which  measures  are  common  across  programs 

•Used  M&A  Working  Group  to  down  select  those  to  add  to  the  company 

standard  set  of  measures 


Programs  are  represented 
on  M&A  WG 


Final  update  approved  by 
Enterprise  Process  Board 


Enterprise  Process  Integration  > 

EPi 
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Summary 


*  Deployed  Measurement  and  Analysis  (M&A) 
process  to  all  programs 

*  Streamlined  Measurement  and  Analysis  planning 
for  the  programs 

*  Standardized  M&A  Planning  across  the  programs 

*  Collected  measurement  usage  across  the 
company 

*  Updated  the  set  of  standard  measures  based  on 
results 


Enterprise  Process  Integration  > 

EPi 
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Contact  Information 


•  William  H.  Golaz 

-  LM  Fellow 

-  Lockheed  Martin  Aeronautics 

-  817-935-5701 

-  William.h.golaz@lmco.com 


Enterprise  Process  Integration 


Enterprise  Process  Integration 

ERf 
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Background 


NORTHROP  GRUMMAN 


•  Most  CMMI  adopters  continue  to  write  plans  the  traditional 
way  --  hundreds  of  pages  long,  filled  mostly  with  boilerplate 

•  This  approach  is  not  consistent  with  the  CMMI  model,  and 
makes  the  plans  difficult  (and  time-consuming)  to  create, 
use,  and  maintain 

•  This  presentation  will  describe  a  simple,  easy  to  use 
method  for  creating  a  short  (less  than  10  pages),  CMMI- 
compliant  project  plan 


Topics 


NORTHROP  GRUMMAN 


•  CMMI  planning  practices  (PP,  IPM,  GP  2.2) 

•  Policy,  plans,  process  descriptions,  procedures  -  what’s 
the  difference? 

•  A  10-page  (or  less)  planning  template 


Lessons  learned 


CMMI  Planning  Practices 


NORTHROP  GRUMMAN 


•  Planning  is  mentioned  both  as  a  process  area  and  as  a 
generic  practice  in  all  process  areas 


Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organization  Process  Focus 

Organization  process  definition 

Organizational  Training 

Integrated  Project  Management 

Risk  Management 

Decision  Analysis  and  Resolution 

Organizational  Process  Performance 

Quantitative  Project  Management 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

GP  2.1 

Establish  an  Organizational  Policy 

br1  z.z 

Plan  tne  Process 

GP  2.3 

Provide  Resources 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Manage  Configurations 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10 

Review  Status  with  Higher-Level  Management 

GP  3.1 

Establish  a  Defined  Process 

GP  3.2 

Collect  Improvement  Information 

What  Might  Be  Included  in  a  Plan? 


NORTHROP  GRUMMAN 


GP  2.2  Plan  the  Process 

Establish  and  maintain  the  plan  for  performing  the  process. 


The  plan  for  performing  the  process  typically  includes  the  following: 

•  Process  description  GP  3.1 

•  Standards  and  requirements  for  the  work  products  and  services  of  the  process 

•  Specific  objectives  for  the  performance  of  the  process  (e.g.,  quality,  time  scale,  cycle 
time,  and  resource  usage) 

•  Dependencies  among  the  activities,  work  products,  and  services  of  the  process 

•  Resources  (including  funding,  people,  and  tools)  needed  to  perform  the  process  GP  2.3 

•  Assignment  of  responsibility  and  authority  GP  2.4 


Training  needed  for  performing  and  supporting  the  process 
Work  products  to  be  controlled  and  the  level  of  control  to  be  applied 

Measurement  requirements  to  provide  insight  into  the  performance  of  the  process,  its 
work  products,  and  its  services 

Involvement  of  identified  stakeholders 

Activities  for  monitoring  and  controlling  the  process 

Objective  evaluation  activities  of  the  process 

Management  review  activities  for  the  process  and  the  work  products 
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CMMI  Planning  Process  Areas 


NORTHROP  GRUMMAN 


Project  Planning 

SG  1  Establish  Estimates 
SP  1.1  Estimate  the  Scope  of  the  Project 
SP  1.2  Establish  Estimates  of  Work  Product 
and  Task  Attributes 
SP  1.3  Define  Project  Lifecycle 
SP  1.4  Determine  Estimates  of  Effort  and 
Cost 

SG  2  Develop  a  Project  Plan 
SP  2.1  Establish  the  Budget  and  Schedule 
SP  2.2  Identify  Project  Risks 
SP  2.3  Plan  for  Data  Management 
SP  2.4  Plan  for  Project  Resources 
SP  2.5  Plan  for  Needed  Knowledge  and 
Skills 

SP  2.6  Plan  Stakeholder  Involvement 
SP  2.7  Establish  the  Project  Plan 

SG  3  Obtain  Commitment  to  the  Plan 
SP  3.1  Review  Plans  that  Affect  the  Project 
SP  3.2  Reconcile  Work  and  Resource  Levels 
SP  3.3  Obtain  Plan  Commitment 


I  integrated  Project  Management 

SG  1  Use  the  Project's  Defined  Process 
SP  1.1  Establish  the  Project's  Defined 
Process 

SP  1.2  Use  Organizational  Process  Assets 
for  Planning  Project  Activities 
SP  1.3  Establish  the  Project's  Work 
Environment 
SP  1.4  Integrate  Plans 
SP  1.5  Manage  the  Project  Using  the 
Integrated  Plans 

SP  1.6  Contribute  to  the  Organizational 
Process  Assets 

SG  2  Coordinate  and  Collaborate  with 
Relevant  Stakeholders 
SP  2.1  Manage  Stakeholder  Involvement 
SP  2.2  Manage  Dependencies 
SP  2.3  Resolve  Coordination  Issues 


Other  CMMI  Planning  Implications 


NORTHROP  GRUMMAN 


•  Words  like  “designated”,  “identify”,  “select”  in  other  practices 
imply  a  choice  to  be  made  in  planning 

•  GP  2.6  Place  designated  work  products  of  the  process 
under  appropriate  levels  of  control. 

•  GP  2.7  Identify  and  involve  the  relevant  stakeholders  of  the 
process  as  planned. 

•  PPQA  SP  1 .1  Objectively  evaluate  the  designated 
performed  processes  against  the  applicable  process 
descriptions,  standards,  and  procedures. 


How  do  Plans  and  Process  Descriptions  Differ? 


NORTHROP  GRUMMAN 


Plan 

Process  Description  qp  3.1 

•  Process  roles 

•  Description  of  activities 

•  Applicable  process  and  product 

•  Resources  (including  GP  2.3 

standards 

funding,  people,  and  tools) 

•  Applicable  procedures,  methods, 

•  Schedule 

tools,  and  resources 

•  Assignment  of  responsibility  GP  2.4 

•  Process  performance  objectives 

and  authority 

•  Entry  criteria 

•  Inputs 

At  Level  2,  plans  describe  what  to  do 

•  Product  and  process  measures  to 

be  collected  and  used 

At  Level  3,  the  existence  of  a  process 

•  Verification  points  (e.g.,  peer 

description  means  that  plans  become 

reviews) 

much  shorter 

•  Outputs 

■  Focus  is  on  instantiating  the 

•  Interfaces 

process  (e.g.,  how  often  a 

•  Exit  criteria 

process  executes) 
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A  Top-Level  Comparison 


NORTHROP  GRUMMAN 


Policy 


High-level  “what"  to  do 
(organizational  guidance) 


Process  High-level  “how"  to  do 

(organizational  standard,  tailored  by  projects) 


Procedure  Low-level  “how"  to  do 

(details  needed  to  follow  a  strategy) 


I  nstantiation  of  the  process 
(how  often,  when,  etc.) 


Documenting  Choices  in  Plans 


NORTHROP  GRUMMAN 


•  Policies  identify  what  must  happen 

•  Process  descriptions  and  procedures  describe  the  steps  to 
be  performed 


Plans  describe  how  the  process  is  instantiated 


Bottom  Line 


NORTHROP  GRUMMAN 


•  For  Level  3  and  higher  organizations,  the  existence  of  a 
process  description  means  the  typical  “boilerplate”  process 
descriptions  included  in  a  plan  (e.g.,  DOD-STD-2167A) 
can  be  eliminated 


•  Plans  simply  describe  the  instantiation  of  the  process 

-  Who,  how  often,  what  resources 

•  Plans  capture  the  decisions  about  how  to  best  fit  the 
process  to  the  task  at  hand 

-  By  creating  a  short,  table-based  template,  the  decisions  are 
highlighted 
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Plan  Template 


NORTHROP  GRUMMAN 


The  work  products  to  be  controlled  for  each  project  process,  their  level  of  control,  and  the 
control  authority  authorized  to  make  changes  to  the  work  product  are  defined  in  Table  2.10. 
The  levels  of  control  are  defined  in  Table  2.1 1 


Table  2.10.  Work  Products .  Documents .  and  Records 


PROJECT  PROCESS 

WORK  PRODUCT 

LEVEL  OF  CONTROL 

AUTHORITY 

ProjectPlanning 

Project  Management  Plan 

Project 

ProjectManager 

Engineering  Change  Proposals 
(CDRLA017) 

Project 

ProjectManager 

Project  Monitor  and  Control 

Work  Breakdown  Structure 

Project 

ProjectManager 

TechnologyControl  Plan  (CDRL 
AQ01} 

Project 

ProjectManager 

Contract/Funds  Status  Report 
(CDRL  AO  03) 

Project 

ProjectManager 

Cost-Schedule  Status  Report 
(CDRL  AO  04} 

Project 

ProjectManager 

12 


All  CMMI-required  decisions  are  captured  in  the  template 

The  template  can  include  either  blanks  to  fill  in,  “typical”  values  to  be 
reviewed/modified  as  needed,  or  mandatory  values  set  by  the  organization 


Lessons  Learned 


NORTHROP  GRUMMAN 


•  The  template  has  been  a  very  useful  tool  for  explaining 
CMMI  concepts 

•  The  table  approach  encourages  Project  Managers  to  be 
more  conscientious  in  their  process  decisions 

•  Short  plans  make  them  easy  to  use  -  no  more  shelfware 
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Getting  Performance 
From  Process 
Improvement 


■ 


_ 


Natural 


SPi 


SEI Partner 


©2010  Natural  SPI 


Content 


In  Search  of  the  Missing  Link 
Between  Process  and  Performance  3 


What  is  Process  “Improvement”  17 


Improving  Performance 
Through  Process  Improvement  25 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


Answering  the  Mail 


Dr.  Ashton  Carter,  DoD  Assistant  Secretary  AT&L,  advised 
industry  of  a  new  program  with  the  goal  of  increasing  warfighting 
capabilities  by  three  percent  without  proportionately  higher 
budgets. 

NDIA  responded  with  58  recommendations.  Only  my  late  entry 
addressed  CMMI  and  organizational  performance. 

We  can  answer  the  mail,  but  only  by  talking  about  performance, 
not  just  the  CMMI  or  maturity  levels. 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


What  We  Believe 


There  is  a  pervasive  belief  in  the  defense  contracting 
industry  that  the  achievement  of  CMMI  maturity 
levels,  ISO/AS  registrations,  or  DO-178  compliance 
mean  that  we  have: 

□  Improved  our  processes 

□  Improved  organizational  and  project  performance 

□  Improved  the  quality  of  our  products  and  systems 

□  Become  more  effective  and  efficient 


How  many  of  us  ever  ask  the  question:  Is  this  true? 
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Things  That  Used  to  Mean  Something., 


MegaCorp 
CMMI  Level  5 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


Things  that  Use  to  Mean  Something2: 

The  Changing  Process  Improvement  Life  Cycle 


A  Life  Cycle  for  real  Process  and 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


Things  That  Used  to  Mean  Something3 


Hey  Son, 
How's  school 
going/ 


Report  Card 

CMMI 
ML  3 


Great  Dad! 
I  kant  reed 
or  rite,  but 
I  past  the 
test! 
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Perception  and  Hyperbole  Have  Won 


If  the  status  is  Green,  it’s  Green! 

If  your  organization  has  been  rated  Maturity  Level  3,  then 
it’s  Maturity  Level  3. 

Whether  or  not  there  are  facts,  measures,  details,  or 
analyses  supporting  the  perception  is  irrelevant.  Whether 
or  not  there  was  improvement  is  irrelevant.  Whether  or  not 
the  SCAMPI  method  was  even  conducted  is  irrelevant. 

Sadly,  perception  and  hyperbole  have  won. 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


The  Dumbing  Down  of  Facts  and  Measures1 


Why  do  the  hard 
work  of  trying  to 
understand 
measures  or 
measurement  data 


■  Enterprise  tool 
a  Project  tool 


c/) 

<D 


10.0 

9.0 

8.0 

7.0 

6.0 

5.0 


Jul  13  Jul  20  Jul  27  Aug  3  AuglO  Augl7  Aug24  Aug31  Sep7  Sepl4  Sep21  Sep28  Oct5  Oct  12  Oct  19  Oct26  Nov  2  Nov  9  Nov  16  Nov  23  Nov  30  Dec7  Dec  14  Dec  21  Dec  28  Jan  4  Jan  11  Jan 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


The  Dumbing  Down  of  Facts  and  Measures2 


When  we  can  just  look  at  red-yellow-green  charts  that  can  mean 
anything,  nothing,  or  whatever  we  want  it  to  mean. 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


The  CMMI  and  Performance 


CMMI  Performance 


Operational 

Excellence 

Robust  process 
Maturity  Level 


Efficiency 

lean/flexible 

Performance 

measures 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


What  Do  CMMI  Maturity  Levels  Mean?1 

ML  2  You  manage  process  performance,  and  your  performed  processes  are 
consistent  with  the  practices  defined  in  ML  2  (at  CL  2)  in  the  CMMI. 

ML  3  You  manage  process  performance  in  accordance  with  a  defined  process, 

and  your  performed  and  defined  processes  are  consistent  with  the  practices 
defined  in  ML  3  (at  CL  3)  in  the  CMMI. 

ML  4  You  quantitatively  manage  your  processes,  and  your  performed  and  defined 
processes  are  consistent  with  the  practices  defined  in  ML  4  (at  CL  3)  in  the 
CMMI. 

ML  5  You  quantitatively  optimize  your  processes,  and  your  performed  and  defined 
processes  are  consistent  with  the  practices  defined  in  ML  5  (at  CL  3)  in  the 
CMMI. 


Nowhere  in  the  CMMI  does  it  say  that  achieving  a  maturity  level  means 
the  organization  or  projects  have  improved  business  performance. 
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What  Do  CMMI  Maturity  Levels  Mean?2 


When  a  Tier  1  contractor,  achieves  a 
maturity  level,  it’s  touted  it  in  press  releases 
as  reaching  “the  high  bar.”  But  in  sub-tier 
vendor/supplier  selections,  CMMI  maturity 
levels  are  expressed  as  the  “minimum  entry 
criteria,”  a.k.a.,  “the  low  bar.” 


Why? 
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What  Do  CMMI  Maturity  Levels  Mean?3 

The  Economic  Basis  of  CMMI  Commoditization 


□  Intrinsic  value  isn’t 

□  So  then,  what  is  the  perceived  value  of  the  CMMI? 


□  In  economics,  perceived  value  for  commodities  and  assets  are 
assigned  a  proxy  (currency,  shares)  with  which  the 
commodity/asset  is  bought,  sold,  and  traded 

□  So  then,  what  is  the  proxy  for  the  CMMI? 

□  The  proxy  for  the  CMMI  has  become  appraisals  and,  specifically, 
maturity  levels 

□  And  the  SEI  has  set  up  close  to  500  SEI  Partners  as  their  own 
mints,  to  print  as  much  currency  as  they  want 
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What  is  one  of  the  common  consequences  of 
flooding  the  economy  with  currency? 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


How  We  Know  What  Isn’t  So*! 


So,  why  do  we  believe  that  CMMI  maturity  levels  (or 
ISO  registrations,  etc)  represent  improved 
performance? 

□  Because  we  want  to  believe 

□  Because  someone  in  a  position  of  authority  said  so 

□  Because  we  live  in  a  culture  that  cannot  or  does  not 
distinguish  correlation  from  causation 

□  Because  CMMI  maturity  levels  are  much  easier  to  achieve 
than  real  process  and  performance  improvement 

□  Because  of  the  way  the  CMMI  is  sold  to  us 


15  *  Gilovich,  Thomas;  How  We  Know  What  Isn’t  So,:  The  Fallibility  of  Human  Reason  in  Everyday  Life,  The  Press  Free,  1991 
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In  Search  of  the  Missing  Link 
Between  Process  and  Performance 


How  We  Know 
What  Isn’t  So*2 

Successful 

organizations 

Unsuccessful 

organizations 

When  the 
“benefits”  of  CM  Ml 
use  are  presented 
or  published,  only 
a  subset  of  the 
entire  set  of 

Organizations 
that  use  the 
CMMI 

This  is  the 
subset  of 
organizations 
touted  to 
promote  CMMI 
adoption. 

What  is  the  size 

of  this  subset 

relative  to  the 

superset? 

possibilities  are 
addressed. 

Organizations 

What  is  the  size 

What  is  the  size 

that  don’t  use 

of  this  subset 

of  this  subset 

the  CMMI 

relative  to  the 

superset? 

relative  to  the 

superset? 
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What  Is  Process 
“Improvement?” 
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What  is  Process  “Improvement”? 


What  do  You  Think  is  Process  Improvement? 


What  do  you  do  in  your  organization 
that  you  call  "process  improvement?" 
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What  is  Process  “Improvement”? 


What  is  Provably  Not  Process  Improvement 


The  notion  that  simply  documenting  your  processes  and 
procedures  in  a  way  that  reflects  practices  in  the  CMMI  or 
standards  (ISO  9000,  TL  9000,  AS  9100,  DO-178)  constitutes 
“process  improvement”  is  false  because: 

□  Performance  improvement  can  only  be  achieved  in  the 
performance  of  the  process,  not  the  definition  of  process 

□  Even  if  there  was  high  fidelity  between  your  defined 
processes  and  performed  processes,  how  does  a 
adoption  of  “best  practices”  equate  to  the  high  bar  for 
process  performance? 
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What  is  Process  “Improvement”? 


The  Question  Never  Asked^  Why  Are  We  Doing  This? 


Why?  Is  the  most  important,  yet  most  overlooked,  question  to 
ask  and  to  answer.  Before  your  organization  begins  planning  a 
goal  and  measurement  project,  ask  and  answer  these  questions: 


□  What  business  are  we  in? 

□  What  do  we  need  to  improve,  and  why?  (How  do  we  know  we’re 
not  already  doing  something  well  enough?) 

□  If  we  produce  results  -  goals  and  measures  -  will  anyone  value 
or  use  them? 

□  What  is  our  level  of  ignorance?  Do  we  have  the  slightest  idea 
what  we’re  doing?  Are  we  competent? 


The  leadership  question  of  the  Modern  era  was,  "Can  we  ...?"  In  the  post¬ 
modern  era,  that  question  is  now  moot.  The  leadership  question  of  our  era 
is,  "Should  we  ...?"  and,  if  so,  "Why?" 
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What  is  Process  “Improvement”? 


The  Question  Never  Asked,:  The  Pursuit  of  the  False  Goal 


As  leaders,  we  are  still  addressing  “What  can  we  do?”  not 
“What  should  we  do?” 

The  false  goals  pursued  using  the  CMMI: 

□  Maturity  Levels 

□  Consistency  (for  its  own  sake) 

□  Quality  (for  its  own  sake) 

The  pursuit  of  a  CMMI  maturity  level  and  the  pursuit  of 
measurable  performance  improvement  are  not  mutually 
exclusive  (but  we  unconsciously  treat  them  as  if  they  were.) 
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What  is  Process  “Improvement”? 


When  the  “R”  is  the  “Why” 


When  striving  to  achieve  a  performance  goal,  organizations  often  care 
about  a  positive  Return  On  investment  (ROI).  If  we  ask  “why”  enough 
times,  we  will  eventually  understand  the  real  goal.  For  example  ... 


Draft  goal: 

Question: 

Answer: 

Question: 

Answer: 

Question: 

Answer: 

New  goal: 


We  will  standardize  our  processes. 

Why? 

So  that  people  perform  their  work  the  same  way. 

Why? 

Because  then  we'll  reduce  waste  and  rework  associated  with 
"reinventing"  things. 

Why? 

As  an  organization,  we  can  produce  more  deliverables  with 
the  same  resources  if  people  reuse  work  products. 

Improve  productivity  by  n%  annually. 
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What  is  Process  “Improvement”? 


Improving  Defined  Processes 

When  I  improve  my  defined  processes,  some  of  the  improvement 
activities  my  organization  will  do  include: 

□  Create  new  process  representations  or  change  existing  process 
representations  so  that  people  find  them  more  adoptable,  adaptable, 
and  helpful 

□  Make  my  process  representations  more  accessible 

□  Make  my  process  representations  more  intuitive 

□  Make  my  process  representations  easier  to  change  with  changes  in 
the  business 

□  Change  the  process  representations  such  that  people  comprehend 
them  quicker  and  can  use  them  more  effectively  and  efficiently 

□  Lower  the  cost  of  process  definition  and  maintenance 
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What  is  Process  “Improvement”? 


Improving  Performed  Processes 

When  I  improve  my  performed  processes,  some  of  the 
improvements  we  can  make  include: 

□  Reduce  the  amount  of  time  required  to  perform  the  process 

□  Make  the  performance  of  process  count;  make  every  task  or  step  yield 
a  work  product,  or  a  change  of  state  to  a  work  product  (i.e.,  never 
have  activity  that  yields  nothing  but  activity;  activity  +  work) 

□  Make  process  performance  yield  higher  quality  work  products  (but 
only  if  a  quality  goal  is  justified  by  a  higher  business  goal) 

□  Make  the  performance  of  process  something  that  is  natural  and 
intuitive  to  people 

□  Make  process  performance  rewarding  to  individuals 
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Improving  Performance 
Through  Process 
Improvement 


3  Things  You  Can  Change  to  Improve  Performance 


There  are  only  three  things  you  can  change  to  improve 
business  performance: 


Improving  skills,  knowledge,  and  learning  can  affect 
productivity  and  quality. 


Improving  technology  can  affect  delivery  cycle 
time  through  greater  automation. 
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Improving  processes  can  contribute  to  greater 
effectiveness,  efficiency,  and  quality  via  process 
performance. 
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Improving  Performance  Through 
Process  Improvement 


Measuring  What  We  Improve-, 


Measure  what  you 
improve: 

If  we  improve  processes  but 
then  measure  project  or 
product  performance,  our 
measures  are  “muddled”  and 
results  are  speculative. 

When  we  improve  processes 
and  measure  processes,  we 
reduce  the  variability  of 
influencing  factors,  and  our 
measurement  information  is 
more  meaningful. 


Influencing  Factors 


Project  &  Product 
Performance 
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Improving  Performance  Through 
Process  Improvement 


Measuring  What  We  lmprove2 


When  we  establish  performance  improvement  goals  and  then  execute  plan 
to  achieve  those  goals,  we  need  to  know  which  measures  to  collect,  analyze 
and  use  for  different  performance  vectors. 


Project  Performance 

Product  Performance 

Process  Performance 

Cost  (CPI) 

Quality 

Efficiency 

Schedule 

Functionality 

Efficacy 

Earned  Value 

Maintainability  (and  MTBF) 

Billable  :  Overhead  ratio  (or 

Cycle  time 

Safety 

Direct :  Indirect) 

Scope  management 

Security 

Unit  Product  Cost 

Shelf-life 

Market  life 

Productivity 

T  ailorability/Scalability 
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Improving  Performance  Through 
Process  Improvement 


Contextual  Performance  Measurement 


When  we  talk  about  performance,  or  report  performance  measures  -  cost,  schedule, 
quality,  or  content  -  it  is  almost  meaningless  to  talk  about  one  performance 
dimension  without  addressing  its  relationship  to  the  others. 


Schedule 


Quality 


Poor 


Good 


Content 


Little 


Lots 
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Improving  Performance  Through 
Process  Improvement 


Establishing  the  Link  Between  Process  and  Performance-, 


We  said  earlier  that  simply  documenting  processes  based  on  a 
model  or  standard  is  not  “improvement.”  However,  we  can 
positively  affect  business  performance  dimensions  such  as 
throughput,  efficiency,  efficacy,  and  product  quality  if  the 

following  conditions  are  true: 

1 .  Our  process  representations  (defined  processes  and  assets) 
enable  efficient  and  effective  process  performance,  and  if 

2.  There  is  high-fidelity  between  the  performed  process  and  the 
defined  process,  and  if 

3.  The  affects  of  skill  or  technology  changes  on  performance 
are  negligible 
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Example 


Improving  Performance  Through 
Process  Improvement 


Establishing  the  Link  Between  Process  and  Performance., 


Process  Improvement 

Expected  Business  Performance 
Improvement 

Relationship 

Improve  project  risk  management 
practices 

Projects  perform  within  estimated 
budgets 

Effective  risk  management  practices 
can  mitigate  the  impact  of  risks  that 
cause  over-spending 

Define  test  procedures 

Reduce  the  number  and  density  of 
defects  in  released  products  and 
subsequently  increase  customer 
satisfaction 

Missed  or  incorrectly  performed  steps 
in  testing  causes  product  defects  to  go 
undetected 

Develop  a  standard  for  product 
requirements  and  use  that  standard  to 
review  requirements 

Reduce  the  cost  associated  with  rework 
in  product  design  and  development 

Design  and  development  rework  has 
been  attributed  to  poor  requirements 

Introduce  standards  for  conducting 
meetings 

Reduce  operating  costs 

There  is  waste/loss  cost  associated 
with  meetings  that  do  not  result  in 
defined  decisions,  actions,  or  outputs 

Implement  an  organization-wide 
lessons  learned 

Reduce  operating  costs 

There  is  signficant  waste/loss  cost 
associated  with  individuals  and  teams 
relearning  things  that  have  been 
previously  learned  by  others 

31 


©2010  Natural  SPI 


Improving  Performance  Through 
Process  Improvement 


Process  and  Performance  Manifesto 


To  better  ensure  process  improvement  results  in  business 
performance  improvement: 

□  Either  let  go  of  or  scrutinize  your  existing  beliefs  about  the  relationship 
between  process  improvement  and  business  performance 
improvement. 

□  Don’t  tamper  with  the  system  . . .  know  what  to  improve,  know  what  to 
leave  alone,  and  why  in  both  situations. 

□  Improve  what  you  intend  to  measure,  and  then  measure  that  which 
you  improve. 

□  Connect  process  improvement  to  performance  improvement  by 
defining  the  relationships  between  the  two. 

□  Measure  the  effects  of  process  improvement  on  business 
performance  and,  in  doing  so,  learn  how  to  be  honest  with  yourself. 
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Process  and  Performance  Improvement  in  Harmony 


Are  CMMI  maturity  levels  and  performance 
improvement  mutually  exclusive? 

You’re  going  to  spend  hundreds  of  thousands  of 
overhead  (or  millions)  dollars  achieving  a  maturity 
level  anyway,  so  . . . 

Why  not  offset  some  of  that  cost  by  achieving 
savings  through  real  process  and  performance 
improvement? 


You  can  have  both! 
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Improving  Performance  Through 
Process  Improvement 


NSPI  Clients 


Autodesk 


Colix 


NAV^A  I  R 


A  CRANE 

AEROSPACE  ft 
t  MIC  T  RON  ICS 


PEARSON 

Education 


BAE  SYSTEMS  MITRE 


Tcyas 

Instruments 
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Improving  Performance  Through 
Process  Improvement 


Contacts  and  More  Information 


Michael  West 

SEI-Certified  Lead  Appraiser 

Executive  Certification  in  Leadership,  Notre  Dame 

michael@naturalspi.com 

866-648-5508 

www.naturalspi.com 


Real  Process 
Improvement 


A 


SEI  Partner 


National  Defense  Industrial  Association 
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Raytheon 

Customer  Success  Is  Our  Mission 


High 

Maturity 

Heresy! 

Doing  Level  5 
Before  Level  4 
Without  Data? 

Thomas  Lienhard 
17  November  2010 


Copyright©  2010  Raytheon  Company.  All  rights  reserved. 
Customer  Success  Is  Our  Mission  is  a  registered  trademark  of  Raytheon  Company. 


Where  Do  We  Get  Our  High  Maturity 
Knowledge? 


Raytheon 

Missile  Systems 


The  CMMI 

•Upfront  material 
•Infamous  page  80 

Training  Courses 

•CMMI  Overview 

•Understanding  CMMI  High 
Maturity  Practices 

•Six  Sigma 
Conferences 


Experiences 


Appraisals 

•Leads 

•Mini  Team  partners 

Others 

•Consultants 

•“Experts” 


CMMI 


Training  Courses 


Conferences 


Others 


Appraisals 
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Where  Do  We  Get  Our  High  Maturity 
Knowledge? 


Raytheon 

Missile  Systems 


[My]  Evolution  of  High  Maturity 
Understanding 


Raytheon 

Missile  Systems 


Evolution  of  a  High  Maturity  practitioner 


SVJ  Six  Sigma 

B!acf;Be!t 


S\a/  C«  C«  Merstan*!  High 

Maturity  Practices 


l/nt[erstani|  Business 
electives 


peer  analysis  of  peer  review  control 
reviews  variation  defect  density  charts 


identify  what 
REALLY 


matters 

predict  performance 
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Nirvana  at  Level  5? 


Raytheon 

Missile  Systems 


■  Achieved  SW  CMM  Level  5  in  2001 

■  Did  not  see  the  “promised”  8:1  ROI 

■  What  went  wrong? 

■  Is  it  about  finding  an  iterative  process  to  collect 
data  so  SPC  can  be  applied? 

■  Is  it  about  hanging  a  sticker  on  the  wall? 

■  Is  it  about  appeasing  the  SEI  to  avoid  an  audit? 

■  Or  is  it  about  meeting  your  primary  business 
objectives? 

■  Needed  to  understand  our  business 
and  business  objectives 

■  Needed  to  understand  which  processes  had  the 
greatest  impact  on  business  objectives 


Ignorance 

Youll  Never  Know  How  Bad  Things  Are 
If  You  Never  Ask 


When  you  think  you  have  it  right,  talk  with  those  responsible  for  cost  and 

schedule 


Understanding  What’s  Critical  to  Our 
Business 


Raytheon 

Missile  Systems 


Development 


l&T 


Production 


cn 

c 


Q)  1- 
O  3 
C  O 

c 

Lit 


0 


Potential  Savings:  Recurring 


Potential  Savings:  Non-Recurring 


Months 


Figure  1.  Typical  Product  Development  Cycle 


Production  is  where  opportunity  abounds 


I  1/  io/zu  iu 


Changing  Our  Approach 


Raytheon 

Missile  Systems 


How  to  move  from  a  Business  that... 


To  a  Business  that 
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Raytheon 

Missile  Systems 


High  Maturity  Timeline 

Statistical  techniques 


SPC 


Pre-Concept  Planning 
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Development  Manufacturing  Field 


1950-1 970s 


HW 
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Org  Objectives 


Balance  performance  with  producibility  and  affordability 
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Raytheon 

Missile  Systems 


High  Maturity  Timeline 
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High  Maturity  Timeline 

Statistical  techniques 


SPC 
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Balance  performance  with  producibility  and  affordability 
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in  Shift 


Raytheon 

Missile  Systems 


SW  SE  HW 

SW  SE  HW 

SW  SE  HW 

SWSEHW 
SyDe 
SystDeve 

System  Development 


Remember,  What  is  Critical  to  the 
Business 


Raytheon 

Missile  Systems 


■  Production  over  Development 

-  Production  is  where  cost  and  time  are  either  minimized  or  super-inflated 

-  The  organization  is  willing  to  invest  more  in  development  in  order  to  streamline 
production 

■  Production 

-  Software 

•  Hit  control  C 

•  Rarely  impacts  development  decisions 

-  Hardware 

•  Extremely  complex 

•  Very  much  impacts  development  decisions 

■  Primary  focus  is  HW/SE/SW  (System  Development) 

■  The  life  cycle  includes: 

-  pre-concept 

-  development 

-  manufacturing 

-  fielding 


Leveraging  our  capabilities  to  be  innovative,  fast,  and  effective 


A  single  Standard  Missile-3,  produced  by  Raytheon, 
is  launched  from  the  USS  Lake  Erie  cm  Feb.  2Q. 
The  missile  hit  its  target  in  space,  destroying!  a 


potentially  hazardous  rogue  satellite. 


U.S.  Navy  photo  via  ap 


Profound  Shift  in  Focus 
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Profound  Shift  in  Focus 
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Profound  Shift  in  Focus 
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Profound  Shift  in  Focus 


Raytheon 

Missile  Systems 


Development 

sw 

1st 

SE 

2nd 

HW 

2nd 
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Profound  Shift  in  Focus 


Raytheon 

Missile  Systems 


Pre-Concept 

Development 

Production 

Field/Maint 

SWSEHW 
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Profound  Shift  in  Focus 
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Production 

Field/Maint 
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SWSEHW 
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1st 

1st 
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High  Maturity  “Epiphany” 


Raytheon 

Missile  Systems 


SW/SE 


Plan 

Req 

Design 

Imp 

Test 
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High  Maturity  “Epiphany” 


Raytheon 

Missile  Systems 
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Modeling  Throughout  the  Lifecycle 


Raytheon 

Missile  Systems 


Vehicle  Pre  Design  Group 

Feasibility 

•Models 
•Baselines 
•Program  Objectives 
•Subprocesses 
effected/selected 
•Predictions/Analysis 
•Recommendations 
•Improvements 
•Measured  effect 


Models 

Baselines 


Robust  Design  Group 
Performance,  Producibility,  Affordability 


Modeling  &  Simulation  Group 
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Breaking  the  Paradigm  -  Level  5 
Before  4  Without  “Data” 
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Start  with  business  goals  and  drivers 
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Take  a  product  centric, 
multi-discipline  approach 
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Establish  objectives  for  entire  lifecycle 
(pre-concept  through  delivery) 
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MDO  Project  Concept  Demonstration: 
Example  Results 


Some  Tools  used  at  Raytheon  Missile 
Systems  for  Robust  Design 
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Results  Driven  -  Product  Centric  -  Full  Life  Cycle  -  Multifunctional  Approach 
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Business  Objective  - 
Increase  Margin  Profit 
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■  Cost  of  Poor  Quality  is  Too  High 

-  Cost  and  schedule  need  to  be  reduced 

-  Rework,  scrap  and  support  costs  need  to  be  reduced 

Production  Rework  Costs 


Improving  production  yields  greatly  reduces  costs,  schedule,  rework,  and  scrap 
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Case  Study 
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■  Multiple  projects  using  a  common  seeker  have  an  Average  Unit  Production  Cost 
(AUPC)  objective 

■  Sensitivity  analysis  showed  which  subprocess  was  the  significant  cost  driver 

■  EOSPA  predictive  cost  model  was  created  to  characterize  the  process 
performance  based  on  organizational  historical  baseline  data 

■  Prediction  showed  the  current  process  was  incapable  of  achieving  the  AUPC 
objective 

■  Causal  analysis  was  done 

■  Process  was  characterized: 

-  Process  steps 

-  Touch  points/hours 

-  Parts 

-  Effort 

■  Improvements  were  identified  and  implemented 

-  Eliminated  non-value  added  process  steps 

-  Reduced  number  of  touch  points  and  touch  hours 

-  Reduced  cycle  time  and  touch  labor 

-  Eliminated  parts 

-  Substituted  processes  with  new  processes  which 
had  reduced  touch  points/hours 


Baseline  vs  Improved  Process 


Miisile  Systems 


Serialized  operation 
reduction 

Reduced  cycle  time 
and  touch  labor 

Modular  assembly 
operations 
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Predictive  Analysis 
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■  Probability  of  Noncompliance  (PNC):  Probability  of  exceeding  either  lower 
or  upper  specification  limits 

■  Distribution  fitted  PNC  =  30% 

■  Predicted  estimated  cost  of  scrap:  $XXM  Annually 

■  Predictive  Analysis  Revealed: 

-  Out  of  Control  conditions  will  occur 

-  Out  of  Spec  conditions  will  occur 

-  Mean  was  too  close  to  lower  limit  - 

•  need  to  center  the  distribution 

-  Variance  was  too  large  - 

•  identify  /  reduce  sources  of  variation 


Low  yields  predicted  as  a  result  of  poor  process  capability 
180  out  of  600  units  would  be  scrapped 
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Results 
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■  Resulted  in  59%  fewer  process  steps 

■  45%  fewer  parts 

■  44%  less  time 

■  78%  improved  throughput 

■  Predicted  yields  increased  to  over  90% 

■  Predicted  rework  reduced  by  over  50% 

■  Predicted  scrap  reduced  by  over  40% 

■  Initial  inspection  costs  reduced  by  over  50% 

■  Warranty  costs  expected  to  be  reduced  by 
over  25% 

Tech  support?  My  predictive  analysis  is  giving  the 
wrong  answer  again  -  can  you  please  fix  it? 

Discipline,  compliance,  predicted  performance  and  continuous  improvement 
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Summary 
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■  To  meet  business  objectives,  engineering  must  be 
more  predictable 

-  Need  to  characterize  process  and  product  performance 
prior  to  implementation 

-  Need  to  establish  and  track  design  metrics  that  relate  to 
business  objectives  (production  yield  and  cost) 

-  MUST  balance  affordability  and  producibility  as  well  as 
technical  performance 


Design  for  cost  and  producibility  has  become  part  of  our  DNA 
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Questions 
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Contact: 

Thomas 


Lienhard@Raytheon.com 
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®  CMMI  and  CMM  are  registered  in  the  US  Patent  and 
Trademark  Office  by  Carnegie  Mellon  University 
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■  Overview 

■  Why  CMMI®  Maturity  Level  5? 

■  Quality  and  Process  Performance  Objectives 

■  Raytheon  Integrated  Defense  Systems  Return  on  Investment 

■  Summary 
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Overview 
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Why  Maturity  Level  5? 

Where’s  the  benefit? 

It  costs  too  much!  Who’s  paying  for  this? 

Maturity  Level  3  is  all  we  can  afford! 

My  customer  says  I  only  have  to  be  Maturity  Level  3! 

What  is  the  impact  of  Maturity  Level  5  on  business  goals? 

This  presentation  will  summarize: 

•  Why  CMMI®  Maturity  Level  5  offers  value  to  a  business 

•  Benefits  Raytheon  Integrated  Defense  Systems  has  realized 
from  deploying  Maturity  Level  5  processes 
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High  Maturity  Controversy 


Founding  Fathers  Debate  High  Maturity 


sw 

CMM® 


Gansler 

Memo 


CMMI® 


The  Race  is  On! 
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Fallout! 


Raytheon 


“We  have  five  maturity  levels. 
We  know  three  of  them  work.” 

Mark  Schaeffer 

Office  of  the  Under  Secretary  of  Defense 
NDIACMMI  Technology  Conference,  2007 
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Response! 


Criteria  for  Audits  of  CMMI® 
High  Maturity  Appraisals 
Announced! 


And  now...  11 
Frequently 
Misinterpreted 
ML  4-5  Practices! 


/  \ 

SCAMPISM  High  Maturity  Lead 
Appraiser  Certification 
Announced! 

V _  _ J 


(  \ 

If  You’re  Living  the  High 


®  CMMI  is  registered  in  the  US  Patent  and  Trademark  Office  by  Carnegie  Mellon  University 
SM  SCAMPI  is  a  service  mark  of  Carnegie  Mellon  University 
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Why  CMMI®  Maturity  Level  5? 

■  CMMI®  was  created  to  support  business  improvement 

-  CMMI®  is  a  means  to  an  end,  not  an  end  unto  itself 

-  As  a  model  containing  best  practices,  CMMI®  is  a  business  strategy  tool  to  help 
achieve  business  objectives 

■  Business  objectives  often  differ  based  on  an  organization’s  level  of  maturity 

-  ML  2  and  ML3  are  about  avoiding  disasters  and  gaining  control 

•  ML  2  organization  objectives:  avoid  disasters  due  to  unrealistic  plans,  lack  of  requirements 
management,  poor  CM,  management  without  progress  measures,  etc. 

□  Example:  begin  measuring  individual  project  productivity 

•  ML  3  organization  objectives:  increased  consistency  of  performance,  common 
organizational  process  tailored  by  all  projects,  proactive  project  management 

□  Example:  collect  productivity  on  all  projects  using  a  tailored  common  process 

-  ML  4  and  ML  5  are  about  managing  against  quality  and  process  performance 
objectives  to  improve  performance  capability 

•  ML  4  and  ML  5  organization  objective:  quantitatively  manage  ability  to  perform  against 
quality  and  process  performance  objectives  that  align  with  business  objectives,  take  action 
accordingly,  and  perform  causal  analysis  to  improve  performance 

□  Example:  understand  variation  in  productivity,  quantify  ability  to  perform  against  productivity 
objective,  target  corrective  action  to  improve  productivity  at  project  and  organizational  levels 
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CMMI®  Maturity  Level  3  vs.  Level  5 
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Maturity  Level  3 

PMC  SGI:  Actual 
performance  and 
progress  of  the  project 
are  monitored  against 
the  project  plan. 

IPM  SP  1.5:  Manage 
the  project  using  the 
project  plan,  the  other 
plans  that  affect  the 
project,  and  the 
project’s  defined 
process. 

OPFSG1:  Identify 
improvements  to  the 
organization’s  process 
and  process  assets. 


Maturity  Level  5 

OPPSP  1.3  Establish 
and  maintain  quantitative 
objectives  for  quality  and 
process  performance  for 
the  organization. 

QPM  SGI :  The  project  is 
quantitatively  managed 
using  quality  and  process 
performance  objectives. 

OID  SGI:  Process  and 
technology 
improvements,  which 
contribute  to  meeting 

quality  and  process 
performance  objectives, 
are  selected. 
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Quality  and  Process  Performance 
Objectives 
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The  engine  that  drives  The  engine  that  drives 

project  performance  business  performance 


Productivity 


Defect  Density 


Defect  Containment 


Requirements  Volatility 


The  engine  that  drives  high 
maturity 
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IDS  Quality  &  Process  Performance  Raytheon 

Objectives 


Raytheon  Goals 

Be  regarded  as  a  customer 
focused  company. 

Grow  revenue  faster  than 
market.  Build  on  good 
performance  in  improving  cash 
flow.  Execute  well  and  with 
predictability. 

Retain  and  attract  world-class 
talent  while  providing  superior 
opportunities  for  employee 
development.  Treat  all 
employees  with  respect. 

Leverage  our  diversity  efforts  as 
a  competitive  advantage, 
continuing  Raytheon’s  leadership 
in  diversity. 

Improve  ROIC  for  Raytheon 
Company.  Take  R6o™  to  the 
next  level,  further  engaging 
customer  and  partners.  Deliver 
greater  value  and  predictability 
through  IPDS,  EVMS,  and 

CMMI®. 

1 

r 

Raytheon  Integrated  Defense  Systems  (IDS)  Goals 

4 

r 

Raytheon  IDS  Engineering  Quality  and  Process  Performance  Objectives 

Cost  Schedule  Quality  People 

CPI  >  X  SPI  >  X  Defect  Containment  >  X%  Average  X  Hours  Training 

Productivity  X%  >  Bid  Productivity  X%  >  Bid  Defect  Density  <  X  per  Employee 

Defect  Containment  >  X%  On-time  Deliverables  Requirements  Volatility  < 

Requirements  Volatility  <  Average  >  X%  X% 

X% 
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Raytheon  IDS 
Return  on  Investment 
from  Implementing 
CMMI®  Maturity  Level  5 
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Assumptions  - 1 

■  Data  used  in  this  study  compared  IDS  2005  performance  2005  -  2008 

-  IDS  achieved  ML  3  for  SE/SW  in  2003,  ML  4  for  SE/SW  &  ML  3  for  HW  in  2005,  and  ML  5 
for  SE/SW/HW  in  2008 

■  Actuals  and  Estimates 

-  In  most  cases,  actual  data  was  used  in  this  study.  If  actuals  were  not  available, 
conservative  estimates  were  used  with  rationale 

•  Projecting  future  return  from  CAR/O ID  projects  is  generally  estimated 

■  Defining  “Investment” 

-  The  cost  of  all  activities  to  incorporate  maturity  level  4  &  5  practices  into  our 
business  processes  and  be  appraised 

•  Appraisal  costs,  development  and  deployment  of  updated  processes  and 
enablers  (e.g.,  process  performance  models),  and  training 

■  Calculating  “Savings” 

-  Apply  baseline  rates  (e.g.,  2005  productivity)  against  2008  size  (e.g.,  lines  of 
code)  to  calculate  “projected  cost  at  baseline  rates” 

•  Projected  cost  at  baseline  rates  =  “2005  productivity”  *  “2008  size”  *  “2008  labor  rate” 

-  Compare  “projected  cost  at  baseline  rates”  to  2008  actuals 

•  Savings  =  “projected  cost  at  baseline  rates”  -  2008  actuals 
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CAR/OID 
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■  Sample  included  19  CAR/OID  process  improvement  projects 
-  14  CAR,  5  OID 


Total  ROI 

Highest  ROI 

Lowest  ROI 

Median  ROI 

Number  of 
data  points 

ROI  on  CAR/OID  projects 

38.4  :  1 

183.3  :  1 

1.9  :  1 

14.3  :  1 

19 

ROI  on  OID  projects 

57.1  :  1 

183.3  :  1 

10.7  :  1 

50.8  :  1 

5 

ROI  on  CAR  projects 

25.8  :  1 

85.5  :  1 

1.9  :  1 

9.6  :  1 

14 

■  The  large  ROI  on  CAR/OID  projects  is  not  surprising 
-  CAR/OID  projects  focus  on  IDS  Engineering  goals 

-7  Defect  Containment,  10  Productivity,  1  CPI,  1  Measurement  Team  Training 
■OID  ROI  is  much  higher  than  CAR  projects 

■Benefits  seen  across  programs  provides  more  ROI 


Applying  CAR  project  improvements  wider  across  the 
enterprise  would  result  in  more  savings 
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Systems  Engineering 

■  Requirements  Volatility 

-  56.1%  improvement 

■  Requirements  Development  Productivity 

-  14.3  %  improvement 

■  CPI 

-  4%  improvement 

-  CPI  variance  reduction  of  63% 
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Software  Engineering 

■  DCTI  Productivity 

-  65%  improvement 

■  Defect  Containment  (SW  Design  through  SW  Maintenance) 

-  11.6%  improvement 
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Hardware  Engineering 

Mechanical  Engineering  Productivity 

-  25  %  improvement 
Electrical  Design  Productivity 

-  Analog:  33%  improvement 

-  Digital:  56%  improvement 

Defect  Density  for  Drawing  Checking 

-  65%  improvement 
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On  Time  Deliverables 
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100.00% 

99.00% 

98.00% 

97.00% 

96.00% 

95.00% 

94.00% 

93.00% 

92.00% 


On  Time  Deliverables 


Z003 


2004 


2005 


2006 


2007 


2008 


2009 


Year 

2003 

2004 

2005 

2006 

2007 

2008 

2009 

OTD 

95.32% 

96.94% 

98.17% 

99.66% 

99.89% 

99.82% 

100.00% 

Through  May 


On  Time  Deliverables  increase  4.9%  since  2003. 
On  Time  Deliverables  >  99%  since  2006. 
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Return  on  Investment 

Quantified  ROI  from  IDS  CMMI  ®  Maturity  Level  5  activity:  24  :  1 

■  Comparison  with  industry  -  from  “Performance 
Results  of  CMMI®  -Based  Process  Improvement ”, 

August  2006,  CMU/SEI-2006-TR-004 

■  Median  ROI  4  :  1 

■  Lowest  ROI  1 .7  :  1 

■  Raytheon  IDS  24  :  1 
-  Highest  ROI  27.7  :  1 


■  IDS  results  are  a  direct  consequence  of  meaningful  process  improvement 
aligned  with  the  business  and  engineering  objectives 

■  Focus  on  major  savings  items:  productivity,  defect  containment, 
CPI/SPI  produces  real  results  -  the  essence  of  Level  5! 
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Summary 
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■  What  an  organization  gets  out  of  process  deployment  and  CMMI  ® 
appraisals  is  a  reflection  of  what  the  organization  puts  into  it 

-  Organizations  that  focus  on  a  maturity  level  number  and  “minimal  compliance” 
are  unlikely  to  derive  any  benefits 

■  High  maturity,  with  its  focus  on  quality  and  process  performance 
objectives,  puts  organizations  and  projects  in  a  position  to  succeed 

-  OPP  applies  statistics  to  management 

•  Establish  quality  and  process  performance  objectives 

•  Process  performance  models  that  illuminate  potential  problems  in  achieving  objectives 
and  address  real  project  issues  are  invaluable  to  the  organization 

•  Process  performance  baselines  identify  the  ability  of  the  organization  and  its  programs 
to  perform  in  relation  to  their  objectives 

-  QPM  and  CAR  push  organizational  objectives  into  projects’  daily  activities 

•  Makes  the  objectives  real  to  projects,  and  not  just  rhetoric 

•  Helps  projects  more  fully  understand  their  role  in  business  success 

•  Having  organizational/project  goals  in  project  metrics  charts  focuses  project  reaction 
when  objectives  are  not  being 

-  01 D  prioritizes  process  improvements  based  on  their  relationship  to  objectives 
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Summary  (continued) 


Raytheon 


All  Maturity  Levels  offer  benefits 

■  Maturity  Levels  2  and  3  help  prevent  disasters  and  gain  control 

-  Are  not  focused  on  quality  and  process  performance  objectives  as  a  the  driver 
of  process  improvement  activity 

-  Set  a  low  ceiling  on  benefits  of  CMMI®  -  based  process  improvement 

■  Maturity  Levels  4  and  5  manage  ability  to  perform  against  quality  and 
process  performance  objectives  that  align  with  business  objectives,  and 
take  action  accordingly 


A  high  maturity  organization  where  all  individuals 
recognize  their  role  and  responsibility  for  business 
success  is  an  organization  that  is  more  likely  to  achieve 

success. 
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■  For  future  questions  the  presenter  contact  information  is: 


Michael  Campo 

Email:  Michael_J_Campo@raytheon.com 
Phone  number:  978.858.5939 
Raytheon  Integrated  Defense  Systems 
Tewksbury,  MA  01876 
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Presenter  Biography 


■  Michael  Campo  is  a  Principal  Engineering  Fellow  at 
Raytheon  Company,  with  32  years  experience  that 
includes  roles  as  a  software  developer, 
software/system  integrator,  manager,  software 
project  manager,  and  process  group  leader.  As 
process  group  leader  for  Raytheon  Integrated 
Defense  Systems,  Mike  developed  and  deployed 
processes  that  led  to  achievement  of  CMMI 
Maturity  Level  3  in  2003,  Maturity  Level  4  in  2005, 
and  Maturity  Level  5  in  2008. 


■  Mike’s  present  position  is  IDS  Process  Technical 
Director.  He  is  a  certified  CMMI  Instructor.  Mike  is 
a  member  of  the  CMMI  VI  .3  Core  Model  Team,  the 
CMMI  VI. 3  Training  Team,  the  CMMI  Configuration 
Control  Board,  and  the  NDIA  CMMI  Working  Group. 
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Integrated  Defense  Systems 

Overview _ 

■  Raytheon  Integrated  Defense  Systems  (IDS)  introduced  its  2nd  Generation 
of  Process  Performance  Models  (2G  PPMs)  as  part  of  its  CMMI  high 
maturity  journey 

-  First  generation  models  were  developed  to  address  individual  lifecycle  phase-specific 
issues 

•  Designing  for  Quality,  CAIV,  cost  models 

•  Relationship  to  business  and  project  objectives  often  implicit 

-  2G  PPMs  are  used  to  manage  risks  related  to  business  and  project  objectives  (cost, 
schedule,  quality)  throughout  the  project  lifecycle 

•  Relationship  of  models  to  goals  is  explicit  and  enabled  by  model-embedded  mitigation  strategies 

•  Supports  ongoing,  project  grass-roots  effort  to  achieve  business  and  project  objectives 

■  As  2G  PPMs  began  to  proliferate,  sociotechnical  issues  emerged 

-  Variation  of  individual  needs  became  apparent 

-  PPM  questions  related  to  project  context  manifested  themselves  as  a  series  of  requests 
for  additional  models  and  features 

-  Caused  model  development  and  deployment  issues  related  to  model  “packaging” 

■  This  presentation  will  review  the  use  of  “horizontal  segmentation”  as  a 
strategy  to  support  deployment  of  a  “model  family”  that  supports  Raytheon 
business  goals  and  model-user  needs 
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System  Lifecycle  Analysis  Model 
(SLAM)  

■  Initial  2G  Process  Performance  Model 
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■  Models  influence  of  requirements  volatility  and  requirements  /  design 
overlap  on  software  /  hardware  development  cost  performance 

-  Enables  project  risk  assessment  &  sensitivity  analysis  around  the  likelihood  of 
achieving  performance  cost  objectives,  and  the  development  /  deployment  of 
mitigation  strategies. 


■  SLAM  Model  Inputs 

-  Estimated  %  Design  Complete  at  Systems  Requirements  Release 

•  Confidence  Range  (+/-  5,  10,  or  15%) 

-  Requirements  Volatility  Estimate 

•  Best  estimate  based  on  historical  baseline  for  product  line,  process  tailoring,  etc. 

•  Variance  estimates  built  into  model  based  on  historical  actuals 


■  SLAM  Model  Outputs 

-  Projected  Software  /  Hardware  Cost  Performance  (CPI) 

•  Mean,  Standard  Deviation 

•  95%  Upper  &  Lower  Prediction  Interval  Limits 
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System  Lifecycle  Analysis  Model 
(SLAM) 


Raytheon 

Integrated  Defense  Systems 


014  -  w 


A  B  C 

•/o  SW/HW  Detail  Design  Complete  at  Requirements  Release 
|%  Design  Complete  at  Requirements  Release  I  ~ 
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21 

22 

23 
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[SLAM  6.0.2  (Demo) 
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■  Feedback  from  the  SLAM  deployment  included  requests  for  additional 
features 

-  Can  we  predict  requirements  volatility? 

-  Can  we  predict  rework? 

-  Can  we  predict  IVV  performance? 

-  Can  we  predict  likelihood  of  meeting  schedule? 


■  The  Process  Performance  Models  Team  began  considering  how  to  meet  the 
needs  of  different  users 


-  Build  additional  features  into  SLAM? 

-  Create  separate  models? 

-  Bundle  separate  models  into  one  supermodel? 

■  Decision  made  to  create  a  schedule  risk  model 

-  Used  a  SLAM-derived,  similar  look  and  feel  approach 

-  Provided  an  opportunity  for  further  feedback 
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Scheduling  Analysis  of  Variability 
Engine  (SAVE) _ 

■  Can  we  predict  likelihood  of  meeting  schedule? 
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■  Models  influence  of  individual  task  cycle  time  variation  on  our  ability  to 
deliver  on-time  against  defined  scheduling  requirements 

-  Enables  projects  to  statistically  quantify  risk  associated  in  meeting  schedule  and 
perform  sensitivity  analysis 

-  Enables  project  identification  of  process  changes  that  reduce  risk  (model  contains 
potential  mitigation  strategies) 


-  May  be  used  during  planning,  re-planning,  Estimate  at  Complete  (EAC)  activity 


■  SAVE  Model  Inputs 

-  Estimated  individual  task  activity  duration  (typically  defined  in  terms  of  their  shortest, 
most  likely  and  longest  cycle  time  expectations) 

-  Individual  task  activity  predecessors  (defines  which  tasks  feed  which  tasks) 

-  Target  overall  schedule  duration  (deadline) 

-  Number  of  task  activities 


■  SAVE  Model  Outputs 

-  Probability  of  meeting  target  duration 

-  Average  Duration  (Cycle  Time) 

-  95%  Upper  &  Lower  Prediction  Intervals 

-  %  of  time  each  task  is  on  the  critical  path 
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E3  Microsoft  Excel  -  SAVE  2.2  1  100  SEI 
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Post-SAVE 
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■  Feedback  from  the  SAVE  deployment  included  requests  for  further  model 
refinements 

-  Can  we  model  probability  of  achieving  cost  targets? 

-  Can  we  interface  SAVE  with  Microsoft  Project? 

-  Can  we  integrate  cost  and  schedule  probability  models? 

-  Can  we  go  below  cost  and  schedule,  and  model  size  and  productivity? 


■  The  answer  to  all  the  questions  is  “yes”,  but  returns  us  to  the 
model/supermodel  dilemma 


■  Solution:  Horizontal  Segmentation 
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■  In  1986,  Psychophysicist  Dr.  Howard  Moskowitz  was  hired  by  Campbell’s 
Soup  Company  to  increase  market  share  of  Prego  spaghetti  sauce  over  arch¬ 
rival  Ragu. 

-  Prego  sales  lagged  and  Campbell’s  wanted  to  change  the  sauce  and  improve  its 
consumer  appeal 

-  Through  earlier  work  in  the  1970s  with  Pepsi  and  Vlasic  Pickles,  Moskowitz  had 
suspected  that  a  universal  “one  sauce  fits  all”  solution  was  not  the  answer 

-  The  answer  instead  was  in  the  understanding  and  leveraging  of  variability 

-  Moskowitz  had  45  different  sauces  created  and  taste-tested 

•  Results  indicated  that  consumers  had  different  concepts  of  the  perfect  spaghetti 
sauce  that  could  be  segmented  into  groups 

•  Main  preferences:  plain,  spicy,  extra  chunky 

•  No  extra  chunky  sauce  was  marketed  at  the  time 

•  Campbell’s  made  $600M  over  the  next  10  years  on  extra  chunky 

■  Moskowitz’  breakthrough  approach,  known 
as  Horizontal  Segmentation,  revolutionized  the 
spaghetti  sauce,  soft  drink  and  multiple  other 
consumer  markets 
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Horizontal  Segmentation 

■  Horizontal  Segmentation  -  groups  people  by 
preference  patterns 

-  Contrasts  with  more  traditional  social  stratifications  (age, 
income,  gender,  race,  etc.) 

■  Lessons  of  Horizontal  Segmentation 

-  There  is  no  perfect  sauce.  There  are  perfect  sauces. 

-  There  is  no  perfect  pickle.  There  are  perfect  pickles. 

-  There  is  no  perfect  process  performance  model.  There  are 
perfect  process  performance  models. 

■  Creating  a  suite  of  process  performance  models 
tailored  to  meet  user  preferences  can  facilitate 
deployment 

-  Offering  users  multiple  model  options  empowers  users 

-  Enables  model  selection  based  on  innate  primal  sensibilities 
(freedom  of  choice,  personal  preferences,  contextual  project 
differences) 
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PPM  Horizontal  Segmentation 


Users  of  process  performance  models 
want  options  that  support  personal  or 
project  preferences 

-  Combinations  of  cost,  schedule,  quality, 
integrated  with  other  project  tools 


Based  on  feedback  and  the  concept  of 
Horizontal  Segmentation,  Raytheon  IDS 
has  created  a  cost  &  schedule  model 
suite  derived  from  SLAM  and  SAVE 

-  Supports  business  goals 

-  Address  individual  project  objectives,  risks, 
and  preferences 


Raytheon 

Integrated  Defense  Systems 


Page  1 1 


SAVE+ 


Can  we  integrate  SAVE  with  Microsoft 
Project? 

-  Uses  a  Crystal  Ball  interface  to  MS  Project 

-  Similar  to  Risk+ 
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SLAM 


11/18/2010 


Page  12 


Raytheon 

Integrated  Defense  Systems 


SAVE+ 
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Predictive  Cost  Modeling  (PCM) _ 

■  Can  we  model  probability  of  achieving  cost  targets? 

■  PCM  models  the  influence  of  individual  cost  element  variation  on  our 
ability  to  meet  cost  targets  /  budget  requirements 

-  Enables  projects  to  statistically  quantify  risk  associated  in  meeting  cost  targets  /  budgets 
and  perform  sensitivity  analysis 

-  Enables  project  identification  of  process  changes  that  reduce  risk 

-  May  be  used  during  planning,  re-planning,  EAC  activity 

■  Predictive  Cost  Model  Inputs 

-  Estimated  mean  cost  for  each  individual  cost  element 

-  Associated  expected  range  of  performance  for  each  individual  cost  element  (min  /  max 
for  the  default  triangular  distribution) 


■  Predictive  Cost  Model  Outputs 

-  Predictive  average  cost 

-  95%  Upper  &  Lower  Prediction  Intervals 

-  Probability  of  meeting  cost  target  /  budgets 

-  Total  cost  estimate  per  task 


-  Overall  cost  target 
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■  Can  we  integrate  cost  and  schedule  probability  models? 

■  IPCS  integrates  the  Schedule  Analysis  of  Variability  Engine  (SAVE)  and  the 
Predictive  Cost  Model  (PCM) 

-  Enables  projects  to  statistically  quantify  risk  associated  in  meeting  cost  &  schedule 
targets  and  perform  integrated  sensitivity  analysis 

-  Enables  project  identification  of  process  changes  that  reduce  risk 

-  May  be  used  during  planning,  re-planning,  EAC  activity 


IPCS  Cost  Model  Inputs 

-  Estimated  mean  cost  and  duration  expectations  for  each  individual  element 

-  Individual  task  activity  predecessors  (defines  which  tasks  feed  which  tasks) 

-  Associated  expected  range  of  performance  for  each  individual  element  (min  /  max  for 
the  default  triangular  distribution) 

-  Overall  cost  &  schedule  targets 

IPCS  Cost  Model  Outputs 

-  Probability  of  meeting  cost  &  schedule  targets 

-  Predictive  mean  expectation  for  cost  &  schedule 

-  95%  Upper  &  Lower  Prediction  Intervals 

-  %  of  time  each  task  is  on  the  critical  path 

-  Total  cost  estimate  per  task 


Integrated  Predictive  Cost  & 
Scheduling  (IPCS) 
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■  Can  we  go  below  cost  and  schedule,  and  model  size  and  productivity? 

m  ESP  models  the  influence  of  size  and  productivity  variability  on  our  ability 
to  meet  cost  targets  /  budget  requirements 

-  Enables  projects  to  statistically  quantify  risk  associated  in  meeting  cost  targets  /  budgets 
and  perform  sensitivity  analysis 

-  Enables  project  identification  of  process  changes  that  reduce  risk 

-  May  be  used  during  planning,  re-planning,  EAC  activity 

■  Size  and  Productivity  Cost  Model  Inputs 

-  Estimated  mean  size  and  productivity  for  each  individual  cost  element 

-  Associated  expected  range  of  performance  for  each  size  and  productivity  estimate  (min  / 


■  Size  and  Productivity  Outputs 

-  Predictive  average  cost 

-  95%  Upper  &  Lower  Prediction  Intervals 

-  Probability  of  meeting  cost  target  /  budgets 

-  Total  cost  estimate 


max  for  default  triangular  distribution) 
-  Overall  cost  target 
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■  Stakeholder  groups  and  projects  have  found  the  family  of  developed  2G  PPMs 
both  easy  to  use  and  conceptually  aligned  with  project  issues. 

■  Projects  have  identified  and  implemented  specific  improvements  as  a  direct 
result  of  the  integrated  deployment  of  2G  PPMs  that  have  enabled  their 
execution.  Resulting  improvement  efforts  include: 

-  Increased  up-front  investment  in  integrated  Engineering  planning  &  analysis 

-  Process  Performance  trade  studies 

-  Process  redesign 

-  Enhanced  peer  reviews 

-  Resource  reallocation  and  conflict  resolution 

■  In  addition  to  deployed  projects,  2G  PPMs  have  been  used  up-front  during  the 
bid  &  proposal  phase  and  are  utilized  by  Engineering  Management  during 
schedule  negotiations  with  program  management. 

■  Significant  qualitative  benefits  from  integrated  deployment  of  these  models 
cannot  be  underestimated  as  projects  leads  /  teams  are  clearly  thinking  and 
behaving  differently  with  respect  to  their  analysis  of  risk  &  opportunity. 
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Summary 

■  Oddly  enough,  while  project  practitioners  may  tend  to  overestimate 
contextual  differences  (your  model  does  not  apply  to  me  because  my 
project  is  different!),  analysts  may  tend  to  underestimate  contextual 
differences  and  preferences  in  search  of  universal  solutions  that  may 
or  may  not  exist. 

■  Leveraging  the  concept  of  Horizontal  Segmentation  enables  our 
development  and  deployment  of  Cost  &  Schedule  Risk  Analysis 
Models  by 

-  Sharpening  our  focus  on  understanding  and  serving  our  projects 

-  Enabling  our  understanding  of  contextual  differences  between  differing 
product  types,  project  challenges  and  personal  preferences 

-  Increasing  project  team  understanding  of  the  benefits  of  statistical  modeling 
and  igniting  their  passion  for  more... 

-  Reinforcing  the  need  for  data  stratification 

-  Challenging  and  reinventing  our  existing  proposal  /  project  management 
measurement  &  analysis  processes  and  tools 


“There  is  no  perfect  Process  Performance  Model. 
There  are  perfect  Process  Performance  Models.” 
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and  User  Group,  November  2009. 

4.  Malcolm  Gladwell  on  spaghetti  sauce, 
http://www.ted.com/talks/lang/eng/malcolm_gladwell_on_spaghetti_sauce.html 
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Michael  Campo 

Email:  Michael_J_Campo@raytheon.com 

Phone:  978-858-5939 

Raytheon  Integrated  Defense  Systems 

Neal  Mackertich 

Email:  Neal_A_Mackertich@raytheon.com 

Phone:  781-440-2629 
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Do  You  Need  CMMI? 


Recognize  these  symptoms? 

•  Missed  commitments 

-  Late  delivery 

-  Last  minute  crunches 

-  Spiraling  costs 

•  Inadequate  management  visibility 

-  Too  many  surprises 

•  Quality  problems 

-  Too  much  rework 

-  Functions  not  working  correctly 

-  Customer  complaints 

•  Poor  morale 

-  Crisis  atmosphere 

-  High  turnover 

-  Low  productivity 
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Does  the  following  occur? 

•  Poor  planning 

-  Plans  not  realistic  or  followed 

-  Work  is  not  tracked  against  the 
plan;  plans  are  not  adjusted. 

•  Baselines  not  controlled 

-  Inconsistent  requirements 

-  Changes  not  managed 

•  Ineffective  organizational  structure 

-  Functions  not  well  integrated 

-  Designs  not  producible 

•  Unable  to  repeat  successes 

-  Staff  skills  and  knowledge  not 
available  when  needed 

-  Dependent  on  heroic  individuals 


CMMI  Features  Help  Address  Common  Issues 


STRENGTH  TIIKOIGH  1M1LSTKV  &  TECHNOLOGY 


CMMI  Feature 

Description  and  Examples 

Results  Oriented 

•  Industry  best  practices  for  project  planning  and  execution 

•  Performance-driven  measures  for  consistent  outcomes 

Priorities  Based  on 
Business  Value 

•  Investments  and  maturity  prioritized  to  align  with  business  goals 

•  Appraisals  relative  to  model  to  set  direction  (“map  and  compass”) 

Customer  Focus 

•  Validation  of  customer  needs  across  the  project  life  cycle 

•  Manage  product/service  quality  (verification,  validation,  reviews) 

Proactive 

Management 

•  Forward-looking  measurement,  monitoring,  risks,  corrective  action 

•  Management  decisions  based  on  plans,  data,  alternatives 

Flexibility 

•  Adaptable  to  a  variety  of  businesses  (domain,  size,  products) 

•  Non-prescriptive  (required,  expected,  informative  components) 

Business  Process 
Integration 

•  Cross-functional  stakeholder  involvement 

•  Coordinate  various  improvement  strategies  and  methods 
(Lean,  Six  Sigma,  ISO,  Agile,  etc.) 

Continuous 

Learning 

•  Standardized  assets  tailored  for  project  characteristics 

•  Leverage  experience  and  history  across  projects 

CMMI  for  Executives 
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Why  Focus  on  Process? 
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PEOPLE 


The  quality  of  a  system  is  highly  influenced  by  the  quality  of  the 
process  used  to  acquire,  develop,  and  maintain  it. 

•  A  long-standing  premise  in  manufacturing 

•  Good  processes  increase  the  likelihood  of 
successful  projects 

Process  can  enhance  the  capabilities 
of  your  workforce 

•  Work  smarter,  not  just  harder 

•  Leverage  organizational  experience 
and  best  practices 

Process  integrates  technology 
with  resources 

•  Technology,  by  itself,  will  most 
likely  not  be  used  effectively 


TECHNOLOGY 
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What  Is  CMMI? 


STRENGTH  TIIKOIGH  IMIISTRV  &  TECHNOLOGV 


CMMI  is  a  model  representing  a  collection  of  best  practices  proven 
effective  in  industry 

•  A  framework  for  developing,  improving,  and  sustaining  business  performance 

•  Provides  a  process  focus  on  work  activities 

•  Developed  by  industry  (commercial  and  defense),  government,  academia 

CMMI  targets  three  primary  environments: 


•  Development  - 

Engineering  a  product  or  service 

•  Services  - 

Providing  services 

•  Acquisition  - 

Acquiring  products  and  services 

The  CMMI  product  suite 
consists  of: 

•  Models  and  primers 

•  Appraisal  methods 


CMMI-DEV 

\ 

o 

CMMI  Model 

i 

Framework 

i 

to 

< 

n 

Z _ / 


•  Training  courses 


Capability  Maturity  Model  Integration  (CMMI®) 
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What  CMMI  Can  Add  to  Your  Organization 
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•  Integration  of  business  processes  across  functions  based  on  industry 
best  practices 

•  Visible  project  and  organizational  measures  aligned  with 
achievement  of  business  objectives 

•  Commonly  accepted  process  framework  for  inter-company 
coordination  and  competitor  benchmarking 

•  Repeat  project  successes  through  standardization,  tailoring,  and 
capture  of  organizational  process  assets 

•  Avoid  project  performance  issues  through  process  discipline, 
proactive  management,  and  early  stakeholder  engagement 

•  Predictable  project  performance,  with  fewer  surprises 
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CMMI  Model  Overview 
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Process 

Areas 

Clusters  of  related  practices,  in  several  categories 
•Project  Management  -  planning,  monitoring,  suppliers,  risk,  ... 
•Support  -  CM,  QA,  measurement,  decision  analysis,  ... 

•Process  Management  -  organizational  processes,  training,  ... 
•Engineering  -  requirements,  development,  integration,  ... 

•Services  -  development,  delivery,  transition,  ... 

•Acquisition  -  requirements,  solicitation,  agreements,  ... 

Generic 

Practices 

Enable  process  management,  deployment  and  improvement 

•Plans,  monitoring,  CM,  stakeholders,  objective  evaluation,  ... 

Goals 

Describes  characteristics  for  implemented  processes 

Capability 

Levels 

Achievement  of  process  improvement  within  an  individual 
process  area 

Maturity 

Levels 

Achievement  of  process  improvement  across  a  predefined  set 
of  process  areas  (stages) 
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CMMI  Appraisals 
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Appraisals  compare  organization  and  project  processes 
against  CMMI  models  to  determine  improvement  priorities 

Senior  management’s  role  in  appraisals: 

•  Provide  sponsorship  and  resources 

•  Set  appraisal  scope  and  objectives 

•  Ensure  follow-through  on  appraisal  findings  and  prioritized 
improvement  actions 

CMMI  provides  a  family  of  appraisal  methods,  with  varying 
intent,  confidence  levels,  data  collection,  resources  needed 

•  Flexible  focus:  approach,  deployment,  institutionalization 

•  Rigorous  benchmark  rating  method  (for  maturity  levels) 

•  “Quick  look”  diagnosis  of  process  weaknesses 

Licensed  SEI  partners  deliver  SCAMPISM  appraisal  services 

•  http://www.sei.cmu.edu/partners/directorv/ 

Note  that  for  internal  process  improvement,  company- 
developed  and  other  methods  can  be  effective 


Standard  CMMI* 
Appraisal  Method  for 
Process  Improvement 
(SCAMPIsm  JA, 
Version  1.2: 

Method  Definition 
Document 


SEI  Partner 

CMMI 
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Reasons  You  Should  Adopt  CMMI 
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1.  Increase  customer  satisfaction 

•  Deliver  products  and  services  that  satisfy  user  needs 

•  Deliver  products  and  services  on  time  and  within  budget 

2.  Increase  probability  of  capturing  new  and  repeat  business 

•  Improved  ability  to  meet  commitments 

•  Reduces  customer-perceived  risk  of  award  to  your  organization 

•  Can  be  a  discriminator  relative  to  your  competition 

3.  Increase  profit  through  improved  quality  and  less  rework 

•  Better  predict  actual  costs  through  repeatable  processes 

•  Better  visibility  into  projects  due  to  established  measures  and  analysis  techniques 

•  Significantly  reduce  the  probability  of  problem  programs 

•  Reduce  costs  by  capitalizing  on  organizational  infrastructure,  processes,  training, 
tools  and  early/often  stakeholder  involvement 

4.  Increase  productivity 

•  More  efficiency  through  implementation  of  common  processes,  tools  and  training 

•  Improved  productivity  by  implementing  process  improvement  that  are  directly 
aligned  key  organizational  goals  and  objectives. 

•  Higher  employee  morale  and  less  turnover 
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Benefits  of  CMMI-Based  Process  Improvement 
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Many  companies  cite  performance 
benefits  from  CMMI 


•  Published  in  conferences,  articles, 
papers,  studies,  surveys,  reports 

SEI  collects  quantitative  measures 
of  CMMI  performance  improvement 

•  Technical  reports,  including: 

-  “Performance  Results  of  CMMI- 
Based  Process  Improvement” 
(http://www.sei.cmu.edu/pub/docume 

nts/06 .  reports/pdf/06tr004 .  pdf) 


Performance 

Median 

Category 

Improvement 

Cost 

34% 

Schedule 

50% 

Productivity 

61% 

Quality 

48% 

Customer 

Satisfaction 

14% 

ROI 

4.0  :  1 

CMU/SEI-2006-TR-004. 

Data  from  35  organizations. 
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Industry  Benefits  from  CMMI 


STRENGTH  THROUGH  INDUSTRY  &  TECHNOLOGY 


Benefits  of  CMMI 
Within  the  Defense 
Industry 

Software  Engineering  Institute 
Carnegie-  Mellon  University 
Pittsburgh,  PA  15£13 

May  2010 


So  ft  warn  Engineering  Institute  (jurth^Mrtlnn 


Example  measures  reported  by  NDIA  member  companies: 

Defect  repair  effort 

•Defect  repair  hours:  -58%  (ML3  to  ML5) 
•Defect  cost  savings:  -105  hrs  per  defect 
•l&T  hrs/defect:  -24% 

•Hours/KLOC:  -22%  (ML3  to  ML5) 

Defect  density 

•62%  fewer  high-severity  defects  (ML5) 
•Defect  phase  containment:  +240%  (ML5) 
•>85%  defects  removed  prior  to  sys  test 
•Acceptance  test:  <  0.15  defects/KLOC 

Development  cost 

•SW  development  cost:  -28%  (ML3  to  ML5) 
•Potential  project  savings:  $1 .9M-$2.3M 

Productivity 

•Productivity  gain:  +42%  (ML5,  9yrs) 

Cost/schedule 

•Over  6X  less  likely  cost/schedule  impact 

The  new  data  presented  in  this  report  demonstrates  that  effective 
implementation  ol  good  practices  aided  by  use  of  CMMI  can  improve 
cost,  schedule,  and  quality  performance. 


Benefits  of  CMMI  in  the  Defense  Industry,  Software  Engineering  Institute,  May  2010. 

http://www.sei.cmu.edu/library/abstracts/presentations/CMMI-Benefits-to-Defense-lndustry.cfm 
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CMMI  Adoption 
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CMMI 
appraisals 
are  conducted 
worldwide... 


USA  Non-USA 


Commercial  In-House 
Contractor  for  Military/Government 
Military/Government  Agency 


Qty 

% 

Qty 

% 

425 

35.2% 

3354 

91.8% 

651 

54.0% 

223 

6.1% 

130 

10.8% 

76 

2.1% 

1206 

100.0% 

3653 

100.0% 

...in  a  wide  range  of  businesses 


...in  small 
and  large 
organizations 
and  projects 


>200, 

22.4% 

101-200, 

17.3% 


<100, 
60.2%  < 


>2000,1.6% 
1001-2000,3.3% 
501-1000,5.0% 
301-500,  6.1% 
201-300,  7.3% 

101-200,  17.3% 
76-100,  8.3% 
51-75,  12.6% 


26-50,  22.0% 


<25, 17.3% 


Organization  Size  (Employees) 

(4863  organizations  reporting) 


...at  all  levels  of  process  maturity 


Services  (72.1%) 

•  Business  Services 

•  Engineering  and 
Management  Services 

•  Health  Services 

•  Other  Services 


Manufacturing  (15.7%) 

•  Electronic  and  Electric  Equipt 

•  Transportation  Equipment 

•  Instruments  &  Related  Products 

•  Industrial  Machinery 

•  Other  Mfg  Industries 


Other  (12.2%) 

•  Finance,  Insurance,  Real  Estate 

•  Public  Administration/Defense 

•  Transportation,  Communication, 
Utilities 

Based  on  primary  Standard  Industrial  Classification  (SIC)  codes  reported  in 
CMMI-based  appraisals. 


Contra  ctor  Military/ 


Commercial  for  Military/  Government 
In-House  Government  Agency 


No  Rating  Given 

5.3% 

8.0% 

22.3% 

Initial  (ML1) 

0.6% 

1.4% 

1.0% 

Managed  (ML2) 

25.8% 

31.5% 

45.6% 

Defined  (ML3) 

58.1% 

49.3% 

26.7% 

Quantitatively  Managed  (ML4) 

2.9% 

1.0% 

1.5% 

Optimizing  (ML5) 

7.2% 

8.9% 

2.9% 

(3779  orgs) 

(874  orgs) 

(206  orgs) 

Source:  SEI  Process  Maturity  Profile,  Sept  2010. 
http://www.sei.cmu.edu/cmmi/casestudies/profiles/cmmi.cfm 
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Getting  Value  from  CMMI 

Your  Role  as  an  Executive 
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Set  the  vision  and  direction  for  CMMI-based  improvement 

•  Establish  measurable  objectives 

•  Be  a  visible  sponsor  -  set  expectations  for  involvement 

•  Manage  process  improvement  like  a  project 

Provide  resources  and  support 

•  Funding,  staffing,  tools 

•  Choose  the  best  people  to  lead  -  respected  opinion  leaders 

Keep  it  real 

•  Maintain  relentless  focus  on  business  value  and  program  performance 

•  Involve  projects  and  practitioners  for  the  best  ideas 

•  Hold  people  accountable 

•  Track  and  communicate  progress 

•  Recognize  and  reward  achievement 
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The  Effective  Use  of  CMMI® 
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Summary  of  NDIA  industry  position  statements  for  obtaining  best  value 
from  CMMI  investments*: 


1.  Good  processes  increase  the  likelihood  of  achieving  successful  project  performance 

2.  CMMI  is  a  model,  not  a  standard  -  adapt  CMMI  to  your  business  environment, 
resources,  and  objectives 

3.  Focus  on  business  improvement  objectives  -  a  primary  emphasis  on  achieving 
levels  may  not  achieve  significant  benefits  and  may  increase  rather  than  decrease 
costs 

4.  High  maturity  is  a  business  case  -  justify  the  investment:  many  organizations  find 
business  value  in  improving  processes  even  at  lower  CMMI  maturity  levels 

5.  Maturity  level  ratings  are  not  alone  a  predictor  of  project  performance  -  many 
other  factors  can  be  significant  contributors 

6.  Don’t  specify  maturity  levels  in  acquisitions  -  use  CMMI  to  probe  supplier 
capability  ana  process  execution  risks 

7.  Greatest  benefits  of  appraisals  are  from  improvements,  not  evidence  or  ratings  - 

disproportionate  effort  on  appraisal  preparation  risk  can  diminish  business  returns 


“The  Effective  Use  of  CMMI®”,  NDIA  Systems  Engineering  Division,  June  2009. 
http://www.ndia.ora/Divisions/Divisions/SvstemsEnaineerina/Paaes/CMMI  Working  Group. aspx 
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Want  to  Learn  More  about  CMMI? 
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SEI  CMMI  web  pages: 


What  is  CMMI?  Models 

Conferences  Performance  Results 

FAQs  Background  Information 


CMMI  focus  topics,  guidance,  technical  reports: 


CMMI  and  Agile 
CMMI  in  Small  Settings 
Earned  Value  Management 

Training: 


CMMI  and  Six  Sigma 
CMMI  in  Acquisition 
SW-Only  Organizations 


Process  Improvement  Introduction  to  CMMI 

CMMI  Level  2-3  for  Practitioners  Understanding  High  Maturity 

User  Networks 


SEI  Partner  Network 
Consultants 


Newsgroups,  Blogs,  Wikis 
Conferences 


Questions?  Comments? 


Adoption 

Appraisals 

Contacts 


Product  Line  Practices 
Interpretive  Guidance 
Operations  Organizations 


Intermediate  Concepts  of  CMMI 
SCAMPI  Appraiser  training 


Books,  Periodicals,  Articles 
Asset  Repositories 


Web:  http://www.sei.cmu.edu/cmmi 
Email:  cmmi-comments@sei.cmu.edu 

SEI  Customer  Relations:  (412)  268-5800,  customer-relations@sei.cmu.edu 
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NORTHROP  GRUMMAN 


Get  It  Sold,  Keep  It  Sold 


Making  the  Business  Case  for  High  Maturity 

November  17,  2010 

Richard  L.  W.  Welch,  PhD 
Associate  Technical  Fellow 
Robert  M.  Sabatino 
Six  Sigma  Black  Belt 
Northrop  Grumman  Corporation 


Approved  for  Public  Release:  Northrop  Grumman  Aerospace  Systems  Case  10-1410  Dated  11/3/10 


Warm  &  Fuzzy  Prospects  .  . 


NORTHROP  GRUMMAN 


. .  .  may  initially  attract  management's  attention,  but 
they  will  soon  be  asking,  "What's  in  it  for  me?" 

CMMI  Performance  Results  Summary 


Per  form-iinc  e 
Catesoi-v 

—  la 

Median 

Improvement 

Number  of  Data 
Points 

Lowest 

Improvement 

Highest 

Improvement 

Corn 

34% 

29 

3% 

87% 

Schedule 

50% 

22 

2% 

95% 

Productivity 

61% 

20 

11% 

329% 

Quality 

43% 

34 

2% 

132% 

Customer 

Satisfaction 

14% 

/ 

-4% 

55% 

Poecuru  on  Investment 

4.0-1 

22 

1.7  :  1 

27.7  ;  1 

Note:  The  performance  remlfe  in  i his  rahl Te  express  change  over  \an  ing  periods  of  rime. 


Source:  Gibson,  Goldenson  &  Kost,  “Performance  Results  of  CMMI-Based  Process  Improvement,” 
CMU/SEI-2006-TR-004,  August,  2006. 
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This  Is  Your  Opportunity  to  Market! 


NORTHROP  GRUMMAN 


. . .  How  do  you  make  the  business 
case  for  High  Maturity 


You  Need  a  Structured  Methodology  to  Market  the  Value 
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Getting  Past  the  Warm  &  Fuzzies 


NORTHROP  GRUMMAN 


•  Locate  the  opportunities 

•  Rank  the  candidates  objectively 

•  Plan  to  execute  successfully 

•  Capture  &  market  the  results 

•  Make  it  permanent 


TRP-06-530 


>67%  Decrease 
in  Cost 


1992  1995  1998  2001  2004  2007 


>15x  Increase 
in  Quality 


1992  1995  1998  2001  2004  2007 
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Good  Engineers  Use  Trade  Studies 


NORTHROP  GRUMMAN 


. . .  To  make  the  business 
case  for  High  Maturity 


Organizational  Goals  &  Objectives 
Competing  Resources 
Cost  Constraints 


Use  this... 


Given  this  . . . 

? 


Competing  Improvement  Opportunities 


Identify,  Quantify  &  Prioritize  Improvement  Opportunities 
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Align  Business  Goals  with  the  Work 


NORTHROP  GRUMMAN 


x%  of  Discipline  Process 
Under  Statistical  Control 


Locate  the  Opportunities 
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Choose  the  Right  Projects 


NORTHROP  GRUMMAN 


Criteria 

Evaluator 

Low  Risks 

High  Benefits  1 

\hligh  Org  Impact  1 

Business  Goal 

Alignment 

Voice  of  1 

Customer  (VOC)  \ 

High  Process 

Criticality 

sum 

check 

Director  1 

20 

20 

10 

20 

20 

10 

100 

ok 

Director  2 

10 

10 

20 

10 

20 

30 

100 

ok 

Director  3 

15 

15 

15 

30 

10 

15 

100 

ok 

Director  4 

10 

15 

10 

15 

20 

30 

100 

ok 

Director5 

15 

15 

25 

20 

25 

0 

100 

ok 

Director6 

5 

15 

20 

23 

22 

15 

100 

ok 

Director  7 

10 

20 

15 

25 

15 

15 

100 

ok 

Directors 

5 

15 

10 

20 

25 

25 

100 

ok 

Director  9 

5 

10 

10 

20 

30 

25 

100 

ok 

Director  10 

10 

40 

5 

5 

30 

10 

100 

ok 

Total 

105 

175 

140 

188 

217 

175 

- — §. — 

~~  _  t  l|j 
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Identify  owning 
Directors, 
proposals  and 
evaluation  criteria 


> 


Weight  the 
evaluation  criteria 


> 


Rank  &  score  the 
candidates  to 
determine  the  best 
process 

improvements  to  work 


Rank  the  Candidates  Objectively 
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Plan  Obsessively 
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Identify  data 
to  be 
collected 


evaluate 


Plan  to  Execute  Successfully 
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Success  Is  in  the  Details-  1 
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Artifacts  generated  by 
following  this  process 


Triggering  data  or 
process  condition 


Business  rationale 


NORTHROP  GRUMMAN 

CAUSAL  ANALYSIS  &  RESOLUTION  (CAR)/ORGANIZAITONAL  INNOVATION  &  DEPLOYMENT  (OID) 
PLANNED  ACTIVITES  CHECKLIST 


CAR/OID  ACTIVITY  TITLE 


ORIGINATOR 


INITIATION  DATE 


_ SIX  CATEGORIES  OF  CAR  ARTIFACTS _ 

1 .  Record  of  defects  or  other  problems  analyzed,  and  the  rationale  for  decision  and/or  collect  &  analyze  improvement 
proposals 

2.  Performing  CAR  and/or  OID  &  develop  an  Action  Proposal 

3.  Action  Proposal 

4.  Action  Items  evolving  from  Action  Proposal 

5.  Cost  estimates  of  analysis  &  resolution 

6.  Measures  of  changes  to  performance  resulting  from  resolution. 


1.  CHECKLIST  FOR  RECORD  OF  ANALYZED  DEFECTS  AND  RATIONALE  FOR  DECISION  AND/OR  COLLECT  & 
ANALYZE  IMPROVEMENT  PROPOSALS 

A)  IDENTIFY  RELEVANT  DEFECT  DATA  AND/OR  IMPROVEMENT  PROPOSAL  DATA  TO  BE  COLLECTED 

(Project  operating  according  to  a  well-defined  process  will  systematically  analyze  the  operation  where  problems  still 
occur,  and  implement  process  changes  to  eliminate  root  causes  of  selected  problems) 

1.A.1  □  Customer  reported  defect  or  process  performance  issues 
1.A.2  □  End  user  reported  defect  or  process  performance  issues 
1  .A.3  □  Peer  Review  reported  defect  that  are  systemic  or  significant 
1  .A.4  □  Testing  reported  defects  that  are  systemic  or  significant 
1.A.5  □  Process  Capability  problem 

1.A.6  □  Project  management  reported  problem  that  are  systemic  or  significant 

1.A.7  G  Appraisal  findings 

1.A.8  □  Audit  Reports 

1.A.9  □  Customer  Requests 

1.A.10Q  Employee/stakeholder  suggestions 

1JV.HO  Lessons  learned 

1.A.12Q  Issues  in  measurement  data  that  are  systemic  or  significant 

1.A.130  Quantitative  Management  Results  related  to  significant  special  causes  of  variation 

1.A.140  Proposals  based  on  strategic  direction  (e.g.  AOP  and  SOF  goals  driven  from  Long  Range  Strategic  Plan) 

1.A.150  Proposals  based  on  processes  or  subprocesses  performance  improvements  (e.g.  using  process  performance 

models  (PPMs)  and/or  process  performance  baselines  (PPBs)  to  identify  and  analyze  the  standard  process). 

1.A.16G  Incremental  improvements  from  a  quantitative  understanding  of  the  organizations  performance 
1.A.17D  Other-  Describe: 


B)  SELECTED  METHODjS)  TO  PRIORITIZE  AND  DETERMINE  WHAT  DEFECTS  AND/OR  IMPROVEMENT  PROPOSALS  TO  ANALYZE 
(Determine  impact,  frequency,  cost,  etc.) 

1.B.1  □  Pareto 

1.B.2  □  Histogram 

1.B.3  □  Process  Capability  analysis 

1  .B.4  □  Value  stream  map  (VSM) 

1.B.5  □  Failure  modes  and  effects  analysis  (FMEA) 

1  .B.6  □  Analysis  of  clusters  of  similar  defects  or  problems  from  quantitatively  managed  processes  and 
subprocesses 

1.B.7  □  Use  of  process  performance  models  (PPMs)  to  understand  the  relationship  of  processes  or 

subprocesses  to  higher  level  processes  and  objectives,  and  to  predict  impact  and  calculation  of  cost  benefits 
1.B.8  G  Identify  and  prioritize  based  on  potential  barriers  and  risks  to  deploying  each  proposal 
1  .B.9  G  Consideration  of  how  the  improvement  will  benefit  the  project  over  time  (expected  life  span  of  proposal) 

1.B.10  G  Other  -  Describe:  Reported  defects  in  lessons  learned  by  end  users 
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Document  the  Issue,  Problem  or  Opportunity 
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Success  Is  in  the  Details  -  2 
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Process  performance 
impacts 

Approach  for  root  cause 
analysis 

Root  cause 

Preventive  &  corrective 
actions 


Investigate  the  Root  Cause 
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Success  Is  in  the  Details  -  3 
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Details  of  any  pilot 

Proposal  summary 

Life  cycle  impacts 

Implementation  & 
evaluation  description 

Detailed  action  items 


NORTHROP  GRUMMAN 


3.  ACTION  PROPOSAL 

NOTE:  If  conducting  a  process  improvement  activity  (Organizational  Innovation  &  Deployment)  instead  of  Causal  Analysis  and 
Resolution  (CAR),  it  may  be  required  to  execute  some  of  the  activities  m  steps  3-6  on  a  pilot  first  before  deploying  to  the  organization. 
This  can  be  done  on  a  test  project,  or  through  other  techniques  such  as  simulation  or  modeling  the  improvement.  For  improvement 
proposals  that  may  represent  a  significant,  high  risk,  or  irrevocable  change  to  piocesses  or  tools,  piloting  the  improvement  may  be 
required.  For  proposed  improvements  that  are  conservative,  low  risk,  or  can  be  abandoned  m  favor  of  the  original  process,  piloting  the 
improvement  may  not  be  required. 

DESCRIPTION  OF  THE  PILOT  (Only  applicable  to  OID  efforts  that  require  a  pilot) 

3.A.1  □  Piloting  the  improvement  on  a  test  project 

-  Describe: 


3.A.2  □  Simulating  or  modeling  the  improvement  through  the  use  of  other  statistical  techniques 
-  Describe: 


3.A.3  □  Other 
-  Describe: 


DESCRIPTION  OF  THE  PROBLEM  AND/OR  IMPROVEMENT  PROPSAL 


PHASE  OR  PROCESS  STEP  WHEN  PROBLEM/DEFECT  AND/OR  IMPROVEMENT  OPPORTUNITY  WAS  IDENTIFIED 


DESCRIPTION  OF  ACTION  PROPOSAL  (DOCUMENT  CRITERIA  FOR  EVALUATING  SUCCESS  OR  FAILURE  OF 
CHANGE) 


4.  CHECKLIST  FOR  ACTION  ITEMS  RESULTING  FROM  ACTION  PROPOSAL 


□  CAR/OID  Action  Items  Form  W2-F0Q2  filed  out 
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Develop  the  Action  Plan 
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Cost  &  schedule  to 
implement 

Cost  if  not  implemented 

Measured  performance 
change 

Measured  process 
capability 
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5.  COST  ESTIMATES  OF  ANALYSIS  AND  RESOLUTION 


5.1  □  Time  and/or  cost  for  identifying  and  correcting  defect  and/or  identifying  and  implementing  improvement. 
Provide  details  here: 


5.2  □  Estimated  cost  of  not  fixing  the  problem  and/or  not  implementing  improvement. 


6.  EVAULATE  EFFECTS  OF  CHANGES  -  MEASURES  OF  CHANGES  TO  PERFORMANCE  RESULTING  FROM  RESOLUTION 


A)  MEASURE  CHANGES  IN  PERFORMANCE  OF  PROJECTS  DEFtJED  PROCESS 

(Determine  whether  selected  change  positively  irfouenced  process  performance  and  how  much  / 

6.A.1  □  Change  in  defect  density  (i.e.  change  in  mean  on  control  chart) 

6.A.2  □  Hypothesis  testing,  significance  testing,  or  other  statistical  technique  using  a  before  and  after  process 
performance  baseline  (PPB)  to  determine  if  the  change  is  statistically  significant 
6.A.3  □  Comparing  the  change  to  the  process  performance  model  (PPM)  to  see  if 
predicted  performance  benefits  were  achieved 

6.A.4  □  Use  of  a  PPM  to  determine  rf  the  change  will  positively  contribute  to  meeting  downstream  quality  and 
other  process  performance  objectives 

6.A.5  □  Other  -  Describe: 


B)  MEASURE  CAPABILITY  OF  THE  PROJECTS  DEFINED  PROCESS 

( Determine  whether  selected  change  has  positively  influenced  ability  of process  to  meet  its  quality  and  process- 
performance  objectives  as  determined  by  stakeholders./ 

6.B.1  □  Change  in  ability  of  process  to  stay  within  process  specification  boundaries  or  improved  process 
capability  (e.g.  represented  by  improved  control  limits  on  control  chart.) 

6.B.2  □  Other  -  Describe: 


ORIGINATOR  AND  MANAGER  INITIAL  BELOW  WHEN  PROPOSAL  SET 


ORIGINATOR  INITIALS 


MANAGER  INITIALS 
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Market  the  Results 
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ELECTRO 
;N  FORMA' 


Engineering  PMT 
Steering  Committee 


Meeting  #149 

08  September  2009 


Bob  Tuthill/Joe  Vandeville 

Engineering  Process  Group 
Northrop  Grumman  Corporation 


Brinq  the  Avionics  Source  Contra  Drawinq  . ,  ,  , ,  „  . , 

X-  “  Number  Hours  Per  Melbourne  &  .  .  . 

2008  (SCD)  qeneration  process  under  statistical  _  „  .  ,  „  „  ..  Avionics 

control  with  a  stable  baseline  Source  Control  Draw, ng  Bethpage 

AOP  Cost 

Reduction  - 

74%  Reduction  1 

in  Devlopment  of  14%  BOE 

Source  Control  Reduction 

Drawing  Taken 

Development 

A  A  a  /vj 

<#  <#  ^  ^ 

ng  the  Bench  Test  Procedi 
stable  baseline 


ir  Bench  Melbc 


Bench  Test 
Procedures 


Keep  Management  Aware  of  the  Value  Provided 
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•  Presentations  can  be  found  at  the  DTIC's  NDIA  Conference  Proceedings 
web  site 

-  :tp://www.dtic.mil/ndia/#s2009 


Statistically 
Managing  a 
Critical  Logistics 
Schedule  Using 
CMMI 


November  2007 


NORTHROP  GRUMMAN 


Using  Moving  Average  Models 
^Predict  Process  Performance 


»ni 

7 


m 


November  19,  2009 

Robert  M.  Tuthill 
Six  Sigma  Black  Belt 
Steve  D.  Tennant 
Six  Sigma  Black  Belt 
Northrop  Grumman  Corporation 


NORTHROP  GRUMMAN 

IHI  P  U  T  U  ft  I 


ontrolling  Peer  Reviews  During 
Software  Development 

A  5-Year  Longitudinal  Case  Study 


19  November  2008 


Richard  L.  W.  Welch,  PhD 

Associate  Technical  Fellow 

Steve  D.  Tennant 

SEPG  Lead 
Northrop  Grumman  Corporation 
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Making  It  Grow 
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•  Good 

•  Better 

•  Best 


Document  the  new  process 
Publish  &  deliver  new  training 
Change  your  engineering  rates 


When  Managers  See  Money,  Making  the  Next  Business  Case  Gets 

Easier 
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Making  the  Sale 
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. . .  You  now  can  make  the 


And  you  have  this  . . .  business  case  for  High  Maturity 


Given  this  . . . 


Organizational  Goals  &  Objectives 
Competing  Resources 
Cost  Constraints 


Competing  Improvement  Opportunities 


Results  Sell 
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QUESTIONS 


NORTHROP  GRUMMAN 


Robert  M.  Sabatino 
Northrop  Grumman  Corporation 
(321)726-7629 
Robert.Sabatino@ngc.com 


Richard  L.  W.  Welch,  PhD 
Northrop  Grumman  Corporation 
(321)951-5072 
Rick.Welch@ngc.com 
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Why  to  Measure 


‘Cheshire  Puss,’  she  began,  . . .  'Would  you  tell  me, 
please,  which  way  I  ought  to  go  from  here?’ 

‘That  depends  a  good  deal  on  where  you  want  to  get 
to,'  said  the  Cat. 


T  don't  much  care  where  -’  said  Alice. 

’Then  it  doesn't  matter  which  way  you  go,'  said  the  Cat. 
‘-  so  long  as  I  get  somewhere ,'  Alice  added  as  an 
explanation. 

‘Oh,  you're  sure  to  do  that,'  said  the  Cat,  ‘if  you  only 
walk  long  enough.’ 


“which  way  I  ought  to  go  from  here” 

Bug  Database 


-33000  Records 
With 

36  Attributes 


:  ::::: 
::::: 


:::::  ::::: 


:::::  ::::: 

: 

:  i:i:i 

:  v  v.  .’ 

m 


rr  rr  rr  rr  rr  rr  rr  rrr  rrr  rr  rr  tt  rr  rr  rr  rr  rrr  rr  rr  tt  rr  rr  rr* 


“which  way  I  ought  to  go  from  here” 

Call  Center  -  Calls  Database 


I  Bil-lli  ■  1 1 1  ■  ■  i  BIii-ii  I 


I  ICIi.l  IE.Ii.I  IC.Ihl  ■■■■  U 
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■  ■In  llilli 
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■  ■■■■  . . I-  I 


I  1  ■  ■  |  ■  EMMI 


■  ■■■■  Bi|iilHI>|iil 


mm 


bubbe  k h h 3 a 


Bililn  BUBBE  h  H  3  1 1 

Bililn  1. >1.37  hHBBH 


Bililn  BUBBE  h>iaHa 


m 


mm. 


-45000  Records 
With 

22  Attributes 


Version  View  vs  Other 


Version  vs  QnA'r 


Version  vs  Sub  Module 


Version  vs  Internal 


Version  vs  Cali 


vers  cn  yg  C c r~i |j a n >■ 


mm 


'Immediate’  Level  Analysis 


“That  depends  a  good  deal  on  where  you 
want  to  get  to,'  said  the  Cat.” 


| Count  of  Call  per  View 

Priority 

Company 
Type  of  Call 

Closed  on  Initial  Cali 


Eny1ronmenty5  Internal 


Call  ys  Target  Mileston 


Ca  vs  Environment 


Priority  View  vs  Other 


|view  Cross  vs.  LC  Record 

Company  &  LC 

Priority  &  LC 
Type  of  Ca  1 !  &  LC 

Closed  on  Initial  Cali  &  LC 


Enylronment 


OnAir  Module 


arset  Milestone 


Sub  Module 


OnAir  Module  View  vs  Other 

OnAir  ys  Tar°etM' 

OnAir  ys  Sub 


Internal  Status  View  vs  Other 


Internal  Status  ys  OnAir 


Status  View  vs  Other 


Environment  ys  Company 


| Call  View  vs  Other 

Call  vs  Statu 5 


Ca  vs  nterna  Status 


Call  ys  Company 


Ca  -.f  Sub  Mcdu  e 


Priority  vs  Environment 

Priority  vs  Status 

Prl  o  rltv  vs  I  nte  rn  a  1  Statu  s 


Environment  &  LC 


Version  &  LC 


Go  Live  Target  £  LC 


Target  Milestone  &  LC 


Sub  Module  Ei  LC 


OnAir  Module  &  LC 


internal  Status 


Go  Live  Target 


Interna  1  Status  vs  TargetM;  1 


Statu 5  vs  Sub  Module 


[icom^any  View  vs  Other 

Company  vs  Status 


Company  vs  internal 


Company  vs  Sub  Mcdu  e 


Environment  View  vs  Other 


Environment  vs  Status 


Environment  vs  OnAir 


E  nvl  ro  n  m  e  nt  v s  Ta  rg  etM  1 1 


Environ  me  it;  vs  Sub  Module 


Ca  s  Cn-  i'  Modu  e 


Prl  o  rltv  vs  T a  rg  et  Mile  sto  n  e 


Priority  vs  Sub  Module 


Internal  Status  &.  LC 


Statu  5 


Version 


Internal  Status  vs  Sub 


Statu  5  vs  OnAir 


Status  vs  TargetM!  I 


Company  vs  OnAir 


Company  vs  TargetM!  I 


Priority  vs  Company 


Priority  vs  OnAir  Module 


Priority  vs  Call 


Statu  5  LC 


Utilizing  Project 

Current  ata 

for 

Better  Management 

Decisions 


Increasing  Project  Data  Usability 

Real  Life  Case  Study 


We  have  based  the  presentation  content 
on  the  current  program  raw  data, 
therefore  presentation  accuracy  or  level 
details  presented  may  impacted  by  it 

In  some  cases  we  guesstemate  on  data  or 
some  of  its  segments 


Unit  Improvement  Objectives 

Improve  communication 
among  the  different  stakeholders 

Increase  system  interfaces  management 
and  control  efficiency 

To  increase  insight  to  effort  deviation 
for  better  planning 


Presentation  Objectives 


To  give  the  program  and  the  division  ideas, 
how  to: 

Increase  product  /  deliverable  quality 
Reduce  project  lifecycle  duration 
Reduce  project  cost 
Increase  resource  (human)  utilization 
Increase  processes  efficiency 
Have  better  control  on  effort  distribution 


The  Presentation  Will  Not 


Give  you  detailed  root  cause  analysis 
Will  not  provide  silver  bullets 

Will  not  solve  your  tomorrow  problems 

•  But  it  is  practical  for  next  phase 


~\m* 


Management 

35% 


Specification 

36% 


Development 

29% 


Initial  Effort  Planning 


Effort 


Current  Effort  Distribution 
For  all  Project  Phases 


Total  Effort 

UAT  Design 

096  596 


Implementation 

0% 


Mngt 

2896 


testing 


Construction 

3096 


QA(PCON) 

1% 


Defects  by  originator 


Cancelled  Defect 

796  Defect 


Interface  - 
SPEC  retest 
1% 

interfaces 

396 


SPEC 

11% 


Requirements 

196 


Design 

2496 


Code  retest  &  Canceleld 

196  396 


Defects  by  Type 

Cancelled 

9% 


Documents 


External  Data 

3% 


K-iSSTS 


Program  Modules 


Environments 


Blanks 

73% 


EU  Sales 

096 


Daisy  non-DM 
0% 

CB CAPRI 3  ; 
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Defects  by  Test  Level 


Unittest  (Operation)  Accepttest 
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Let’s  Try  Some  Mix 
and  Match 

We  Will  Demonstrate  How 
Relationships  Between  Measures  Can 
Benefit  the  Organization  for  Better 
Planning  and  Management 


Management  Tool 
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ome  guesstemations  on  cost  effectiveness 


If  an  average  developer  day  cost  is  ~7000units 
The  total  project  effort  was  10022  day  (100%) 

^  The  testing  phase  was  1453  day  (14.5%) 

t3“  Defect  that  are  the  result  of  documentation  are  69%  of  all  defects 


If  we  will  assume  the  to  correct  69%  of  all  defects 
will  take  around  40%  of  the  testing  duration; 

means  that: 
that  will  be  581  day 
With  the  overall  cost  of  4068400units 


However 

Adding  30  review  days  in  the  static  tests 
and  another  80  days  of  code  inspection 
will  end  with  the  cost  of  770000  units 


And  still  we  have  saved  at  least  9401000  units  (1343  days) 
Means  that  we  ware  able  to  reduce  13.04%  of  the  project  time 


• 


• 


•  C33 


What  Organizational  Processes 

we  have  touch 


Scope  and  Size 
Status  meetings 
Static  Tests 

Testing  (planning  and  execution)  all  phases 
Lesson  learned 
Process  Improvement 


CMMI  Effecting  PA’s 

Project  Planning 

Project  Monitor  and  Control 

Measurement  and  Analysis 

Validation 

Verification 

Requirements  Development 
Technical  Solution 
Product  Integration 
Organizational  Process  Focus 


Practical  Improvements 
Suggestions 

Requirements  Development 

•  Writing 

•  Verifying 

•  Validating 

Effort  Distribution 

•  Overhead  planning 

•  Estimation  models 

•  Project  control 

•  Lessons  learned 

Verification 

•  Planning 

•  Guidelines  for  conducting 

•  Checklist 

•  Results  analysis 

•  Efficient  communication 

•  Lessons  learned  and  root  causes 


PHASE 

PERCENT  OF 
EFFORT 

Requirements  Evaluation  Phase 

8% 

Project  Planning  Phase 

3% 

Analysis  Phase 

10% 

Design  Phase 

20% 

Construction  Phase 

32% 

Test  Phase 

23% 

Implementation  Phase 

1% 

Customer  Support  Phase 

2.5% 

Completion  Phase 

.5% 

Characteristic 

Level 

Weightage 

Product  complexity 

High 

1.15 

Main  storage  constraints 

High 

1.06 

Applications  experience 

Low 

1.13 

Programmer  capability 

Low 

1.17 

All  other  characteristic 

Nominal 

1.00 

Effort  Adjustment  Factor 

1.15*  1.06*  1.13*  1.17*  1.00  =  1.61 

Activity 

Small  Project 

Medium  Project 

Large  Project 

User  Documentation 

10 

05 

03 

Project  Management 

25 

15 

10 

Quality  Assurance 

15 

10 

10 

•  JUser  Training 

10 

07 

02 

Acceptance  Testing 

10 

05 

05 

Performance  Tuning 

05 

08 

10 

|!>Tptals  (%age) 
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50 

40 
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IS 


67 
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TOTAL 
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Practical  Improvements 
Suggestions 


Validation 


•  Planning 

•  Guidelines  for  conducting 

•  Checklist 

•  Results  analysis 

•  Efficient  communication 

•  Lessons  learned  and  root  causes 

Measurements 

•  Definition  with  direct  line  to  business  objectives 

•  Measurements  structures,  content  and  context 

•  Guidelines  for  collecting  and  ‘work  with” 

•  Checklist 

•  Results  analysis 

•  Efficient  communication 

•  Lessons  learned  and  root  causes 


BBCWP 


Improve  productivity 


S  BACWP 

^  'so. 


-  BAC 


Computed  Metric  Name 


Control  Measures 


Performance  Measures 


_  Budgeted  Coat  of  Work 
Performed 


Identify  budgeted  labor  hours  associated  with 
the  work  that  has  been  completed. 


How  efficient  are  tests? 


Definition 


Source 


frequency 

dev] 


Monthly 


Monthly 


Monthly 


Defects  detected  through  testing  / 
hour  of  testing 


Defects  detected  through  reviews 
/hour  of  review 


(Actual  size  of  the  product 
delivered  to  the  customer  /  Actual 
effort  spent  to  complete  the 
project]  in  each  technology 
platform 


PINS  (add  size 
fed) 


End  of  the 
project 


KR  artifacts  used  /  project 


How  efficient  are  reviews? 


Question 


Metric 


Testing  efficiency 


Review  efficiency 


What  is  the  productivity  in  fixed 
price  projects? 


Productivity 


How  effective  is  best  practices 
sharing? 


KR  artifact  usage 
index 


Monthly 


KR  artifact 
contribution  index 


KR  artifact  added  /  project 


Alias 


Objective  Df  Computed  Metric 


-  Actual  Coat  of  Work  Performed 


Identify  the  actual  labor  hours  spent  on  the 
project  to  date. 


Id  Budget  at  Completion 


Identify  the  projects  budget. 


Development  &  Interfaces  Integration 

•  Improve  content  of  guidelines  in  the  different  technical  document  to 
build  more  strong  and  clear  descriptions 

•  Peer  reviews 

•  ‘Internal’  documentation 

Quality  Assurance  and  Process  Improvements 

•  Identify  process  goals  and  targets  with  direct  line  to 
business  objectives 

•  Plan  to  process  evaluation;  including: 

Guidelines  for  conducting 
Checklist 

■Results  analysis 

Efficient  communication 
Lessons  learned  and  root  causes 


Practical  Improvements 
Suggestions 


Questions 


o 


Lean  Implementation  of  Organizational 
Process  Focus  (OPF)  and  Risk  Management 
(RSKM)  CMMI®  Process  Areas  in  Agile 
Environment 


Ahmed  Mahdy 

Senior  Software  Engineer  &  Agile  Coach,  Raya  Software. 
SW  Process  Improvement  Consultant 

aamahdvs@qmail.com 
ahmed  mahdv@ravacorp.com 
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Prerequisites 


Basic  background  about: 

°  Agile  Values  &  Principles  (www. 
agilemanifesto.org) 

°  CMMI  version  1.3  (www.sei.cmu.edu/cmmi) 
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Feasibility  of  Agile  and  CMMI 


•  "The  two  methods  are  certainly  compatible"  Mike 

Konrad,  Senior  Member  of  Technical  Staff  Software  Engineering  Process  Management 
Program 

•  "It  was  an  opportunity  for  us  at  SEI  to  dispel  some 
myths  and  'apologize',  in  a  sense,  to  some  people 
in  the  Agile  community”  Mike  Konrad 


•  Why  Not  Embrace  Both  technical  report  by  SEI 


•  Jeff  Sutherland  Recommends  Combining  Scrum 

with  CMMI  Level  5  Jeff  Sutherland  is  one  of  scrum  inventors 


•  And  Others  . .  .  (www) 
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Feasibility  of  Agile  and  CMMI 
(Cont’d) 


•  Our  Own  Various  Implementations 

•  One  of  the  successful  companies  that  applied  Agile 
CMMI  in  the  same  team  &  project  with  light-weight 
processes  is: 

Raya  Information  Technology,  Egypt 

(http://www.ravacorp.com  ) 

RAW 

Information  Technology 
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(Cont’d) 

They  are  Synthetically  welded? 


Permission  of  image  copy  is  down  this  slide 
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(Cont’d) 

But,  they  are 
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How  do  we  look  at  Agile 
CMMI? 


Hdw  ua  do  l:  -aliss  lib  huw  to 
perfomru'conduci  a  process 
through  many  different  and 
unlimited  practices 


Agile  practices,  which  do  rot 
vioteie  Agile  values  antf 
principles 


Agile 

CMMI 

Whfli  lo  do  the  modeJ  of 
CMMI  (Specific  and  Generic 
Practices)  ask£  us  whaL  to 
da  regardless  o1'  the  way  of 
do»ng  it 


N 


CMMI  PA  Goals  (Specific 
and  Genenc  Practices) 


Brief  ReieLidnship  between  AgiSe  and  CMMI  rnun  implerTientrStinn  perspective 
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Why  Agile  CMMI  ? 


Easily  understanding  of  model  intent, 

Thinking  in  terms  of  valuable 
compliance  instead  of  just  compliance  or 
filling  contract  requirement, 

Paying  more  attention  to  people, 

Managed  fast  deliverables. 
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Why  Agile  CMMI  ?  (Cont’d) 

•  Heavy-weight  processes  lead  to: 

°  a  late  delivery, 

°  an  inaccurate  assurance  of  quality, 

°  a  misleading  way  to  the  required  value, 

°  a  more  budget, 

°  a  lot  of  workarounds  by  stakeholders  to 
survive! 

•  Remember,  there  are  some  competitors 
over  there  ! 
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Why  Agile  CMMI  ?  (Cont’d) 


1970.  "Managing  the  Development  of  Large 
Software  Systems:  Concepts  and  Techniques".  In: 
Technical  Papers  of  Western  Electronic  Show  and 
Convention  (WesCon)  August  25-28,  1970,  Los 

Angeles,  USA. 


ANALYSIS 


SYSTEM 

REQUIREMENTS 


SOFTWARE 

requirements 


program 

DESIGN 


Dr.  Winston  W.  Royce 
The  Waterfall  Model 


TESTING 


44  I  believe  in  this  concept,  but  the  implementation 

described  above  is  risky  and  invites  failure. 


OPERATIONS 
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Why  Agile  CMMI  ?  (Cont’d) 


Feature  Usage 


Nev 


Source:  Jim  Johnson  of 
the  Standish  Group, 
Keynote  Speech  XP  2002 


Iways 


Often 


Sometimes 


Rarely _ 

0  Always _ Often _ #  Sometimes _ ♦  Rarely _ >  Never 


Dr.  Sidky:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 


Feature  Usage 


N I # 


45% 


7°/c 


Source:  Jim  Johnson  of 
the  Standish  Group, 
Keynote  Speech  XP  2002 


Iways 


•  We  need  to  focus  more  of  prioritizing  features  and  delivering  value  quicker  than 
focusing  on  implementing  all  the  features 

•We  need  to  focus  on  the  ROI  of  each  feature  for  your  customer  with  the  help  of 
customer 

•We  need  to  focus  on  the  ROI  of  each  feature  as  a  product  manager 


Sometimes 


Rarely _ 

0  Always _ Often _ #  Sometimes _ #  Rarely _ >  Never 


Dr.  Sidky:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 

Challenged  Projects 

m 


USA:  $80-145  billion  per 
year  is  spent  on  failed 
and  cancelled  projects 

UK  :  1 2  out  of  18  Large 
IT  projects  have  failed 


Standish  Report  -  2006 

Dr.  Sidky:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 

However, . . . 

Market  value  of  publicly  traded  shares: 

SNA  (31  December  2009  est.) 
country  comparison  for  the  world 

$34.95  trillion  {31  December  2008) 

$64.56  trillion  <31  December  2007  est.) 


Industries: 


dominated  by  the  onrush  of  technology,  especially  in  computers,  robotics, 


telecommunications,  and  medicines  and  medical  equipment;  most  of  these  advances  take 
place  in  OECD  nations;  only  a  small  portion  of  non-OECD  countries  have  succeeded  in 
rapidly  adjusting  to  these  technological  forces;  the  accelerated  development  of  new 
industrial  (and  agricultural)  technology  is  complicating  already  grim  environmental  problem 


Industrial  production  growth  rate: 


CIA  World  Factbook-  2010 
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Why  Agile  CMMI  ?  (Cont’d) 

However, . . . 

Market  value  of  publicly  traded  shares: 

SNA  (31  December  2009  est.) 
country  comparison  for  the  world 


•What  if  we  increased  our  IT  profits  to  both  vendor  and  client  ? 

•What  if  we  eliminated  (or  decreased)  the  challenged  ,  failed  and  canceled  projects? 


dominated  by  the  onrush  of  technology,  especially  in  computers,  robotics, 


telecommunications,  and  medicines  and  medical  equipment;  most  of  these  advances  take 
place  in  OECD  nations;  only  a  small  portion  of  non-OECD  countries  have  succeeded  in 
rapidly  adjusting  to  these  technological  forces;  the  accelerated  development  of  new 
industrial  (and  agricultural)  technology  is  complicating  already  grim  environmental  problem 


Industrial  production  growth  rate: 


CIA  World  Factbook-  2010 
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What  is  “Agile  CMMI  Practice”  or 
ACP? 

•  Agile  CMMI  Practice  (ACP)  is  the 
practice  (or  way  of  doing  an  activity) 
that  achieves  the  Specific  Practice 
(SP)  in  CMMI  Model  given  that  this 
practice  achieves  at  least  one  of  Agile 
Values  and  Principles  without  violating 
any  of  them. 

Given  that  CMMI  model  defines  the  Alternative  Practice  as  a  practice  that  is 
a  substitute  for  one  or  more  generic  or  specific  practices  contained  in  CMMI 
models  that  achieves  an  equivalent  effect  toward  satisfying  the  generic  or 
specific  goal  associated  with  model  practices.  Alternative  practices  are  not 
necessarily  one-for-one  replacements  for  the  generic  or  specific  practices. 
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Remember:  Agile  Values 

We  are  uncovering  better  ways  of  developing 
software  by  doing  it  and  helping  others  do  it. 
Through  this  work  we  have  come  to  value: 

°  Individuals  and  interactions  over  processes  and 
tools 

°  Working  software  over  comprehensive 
documentation 

°  Customer  collaboration  over  contract  negotiatior 
°  Responding  to  change  over  following  a  plan 

That  is,  while  there  is  value  in  the  items  on 
the  right,  we  value  the  items  on  the  left 

C^^ual  Technology  Conference  and  User  Group 
1  AssocraVror!  ^  ■ 
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In  7  Steps,  how  do  you  achieve 
ACP? 

1.  Make  sure  of  your  senior  management  adoption, 

2.  Discuss  the  policy  and  process  improvement  project  vision 
by  reaching  consensus  with  all  stakeholders  about  the 
vision  and  goals  of  this  project, 

3.  Specify  the  process  areas  in  scope  (i.e.  what?), 

4.  Meet  the  users  (actual  implementers)  of  each  practice  in  the 
selected  process  areas, 

5.  Explore  ,with  the  implementers,  all  possible  valuable  ways 
of  doing  these  practices  which  do  not  violate  any  of  Agile 
Values  and  Principles, 

6.  Consolidate  the  output  from  users  to  reach  the  way  of 
implementing  the  selected  process  areas  and  implicitly  its 
practices  (i.e.  how?) 

7.  Inspect  and  adapt  (i.e.  process  improvement  cycle  is  always 
active) 
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Difference  Between  “Convey” 
and  “Preserve”  Information 


“Convey”  Information 

“Preserve”  Information 

Why? 

Transfer  information  within  your  team  members 
for  alignment  and  having  the  same  understanding 

of  this  information. 

Store  information  for  further  references,  reviews,  reuses 
and  organization  history. 

How? 

It  depends  on  the  case  you  have,  that's  why  Agile 
is  human-centric,  you  choose  the  best  way  for 
conveying  information  in  your  case.  And  do  not 
forget  that  one  of  Agile  principles  is  "the  most 
efficient  and  effective  method  of  conveying 
information  to  and  within  a  development  team  is 
face-to-face  conversation",  however,  if  there  are 
rigid  constraints  in  achieving  this  way,  move  to  the 
next  communication  way  which  invites  interaction 
within  team  members.  This  encourages  you  to 
achieve  the  first  step  of  SAMI  (Sidky  Agile 
Measurement  Index),  it  is  a  5-step  value-based 
roadmap  to  agility  and  designed  to  help  guide 
organizations  seeking  to  become  more  agile.  Each 
of  SAMI's  5  steps  (Collaborative,  Evolutionary, 
Integrated,  Adaptive  and  Encompassing)  aims  to 
instill  a  new  value  in  teams  and  organizations. 

Also,  it  depends  on  the  case.  Usually,  people  prefer  using 
different  types  of  documentations  project  management 
tools,  document  management  systems  and  we  prefer  using 
a  system  that  supports  your  files'  configurations  such  as 
version  history  details. 

When? 

In  every  time  the  team  members  communicate 
any  information  regarding  the  project 

In  all  legal  and  formal  communications.  And  when  you 
want  to  ensure  commitment  and  understanding  of  any 
information  or  responsibilities  with  either  the  customer  or 
project  team  members. 
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What  is  the  worst  thing  you  can 
do  for  your  organization? 


Wrong  implementation  of  CMMI,  an 
implementation  that  leads  to 
appraisal  satisfaction  or 
compliance  regardless  of  the 
values  and  benefits  that  CMMI 


intends  to  provide, 
thing  you  r 
business  or 


is  the  worst 
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Organization  Process  Focus 


•  Purpose: 

The  purpose  of  Organizational  Process  Focus 
(OPF)  is  to  plan,  implement,  and  deploy 
organizational  process  improvements  based  on 
a  thorough  understanding  of  current  strengths 
and  weaknesses  of  the  organization’s 
processes  and  process  assets. 
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Organization  Process  Focus 
(OPF) 

•  Is  it  an  Agile  process  ?  Does  it  support 
any  of  Agile  Principles? 

Yes! 

•  Agile  Principle  #12: 

“At  regular  intervals,  the  team  reflects 
on  how  to  become  more  effective, 
then  tunes  and  adjusts  its  behavior 
accordingly.” 
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Organization  Process  Focus 
(OPF) 

•  Process  Improvement  Main  Activities 

°  Getting  feedback  from  all  team  members 

°  Decide  if  the  feedback  will  be  taken  as  an 
improvement  request,  if  not,  justify  it  with  the  owner  of 
this  feedback 

°  Include  the  feedback  to  Process  Improvement  Live 
Plan 

°  Update  the  process  asset  library  with  the  new 
changes,  and  give  a  training  to  the  teams 

°  Deploy  the  updated  processes 

°  How  you  can  make  it?  It  is  totally  yours! 
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u. 


urgamzauon  rrocess  rocus 


% 

Aajd&Q 


(OPF^ 


Description  :  it  contains  the  action  items  lhat  are 
resulted  from  the  strengths  and  weaknesses 
Attributes.:  Action,  Responsibility.  Implement  lion 
Due  date.  Affected  Processes),  Original  Strength 
or  Weaknesses 


Description:  it  is  just  doing  the  action 
item  and  it  is  expected  that  it  makes  a 
change  in  the  existing  processes  or 
adds  new  processes. 

Do  not  forget:  your  company  vision 
and  policy  while  you  are  going  to  take 
any  process  improvement  action, 
consistency  and  alignment  is  not  an 
option.  Von  could  not  make  it?  return  to 
the  senior  management  and  discuss 
their  vision  and  related  policy  for 
reaching  the  alignment  and  consistency 


Description;  it  is  deploying  the  new/modified 
processes  in  real  or  pilot  projects. 

Do  not  forget  before  deployment:  train  people 
on  the  changes  and  keep  them  updated  and 
synchronized  with  process  improvement  activities 
or  process  asset  Fibrary  changes. 


Organizational  Process  Focus  (OFF)  in  Agile  organizations 

r  Ahmed.  VLuJidv  ujj jtej Jvdv^1  liiTm a  l  I .  c run 
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yr  \ 


urgamzation  process  i-ocus 


-  Feedback  Check  Points 

•  Iteration/Sprint  Retrospective  (freq.1-3 
weeks) 

°  Positives  (or  what  the  team  should  keep 
doing?) 

°  Negatives  (or  what  the  team  should  stop 
doing?) 

°  Ideas  (anything  the  team  wants  to  say  for 
improvement) 

°  Action  Items 

°  Also,  you  can  give  the  training  of  any  new 
change  to  the  team  without  requesting 
trainings!  That’s  why  we  are  coaching 


yr  \ 


#  UJOUUIUIIUI  I 
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urgamzation  process  i-ocus 


-  Check  points  (cont’d) 

•  Release  Retrospective  (freq.  4-6 
weeks) 

°  Positives  (or  what  the  team  should  keep 
doing?) 

°  Negatives  (or  what  the  team  should  stop 
doing?) 

°  Ideas  (anything  the  team  wants  to  say  for 
improvement) 

°  Action  Items  (including  monitoring  previous 
action  items) 

°  Also,  you  can  give  the  training  of  any  new 
change  to  the  team  without  requesting 


CMMI  lOthAnni 
Association 


idustrial 


Risk  Management  (RSKM) 

•  Risk  Management  Activities 

°  Identify  ownership  and  responsibility  of  each 
activity 

°  Identify  risk  and  define  its  attributes  (category, 
probability,  impact,  mitigation,  owner  of  this 
mitigation,  status) 
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Risk  Management  (RSKM)  - 
Check  Points 

■j  •  Project  Team  Kickoff 

\  •  Client  /Project  Kickoff 

|  •  Release  Planning 

|  •  Iteration/Sprint  Retrospective  (freq.1-3 

weeks) 

°  Negatives 

°  Risks  (Add  new  risks,  and  update  the  status  of 
previous  risks) 

°  Action  Items  of  “risk  mitigation”  type 
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Risk  Management  (RSKM)  - 
Check  Points 

■j  •  Release  Retrospective  (freq.4-6  weeks) 

I  °  Negatives 

I  °  Risks  (Add  new  risks,  and  update  the  status  of 
previous  risks) 

°  Risk  mitigation  action  items 
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Risk  Management  (RSKM)  - 
Recommendations 

•  Use  a  tracking  system  for  managing  your 
project/company  risks  and  related  mitigation 
action  items. 

i  •  Radiate,  if  possible,  project  risks  and  related 
mitigation  action  items. 

|  •  Use  constant  colors  in  differentiating  among 
risks’  probabilities  or  impacts  (i.e.  importance) 
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Risk  Management  (RSKM)  - 
Recommendations 


Use  constant  colors  in  differentiating  among 
risks’  probabilities  or  impacts  (i.e.  Risk 
Imp 


Probability 
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Important!  Conclusion 


It  is  not  only  possible  to  embrace  both  agile  and  CMMI,  but  also  it 
is  easier  and  proven  practically.  The  presentation  is  not  supposed 
to  show  best  practices  for  your  organization,  the  fore  mentioned 
practices  may  not  work  in  your  organization. 

However,  inspect  and  adapt  to  reach  the  most  appropriate  practice 
that  adds  the  best  possible  value  to  your  organization  and  your 
client.  Also,  because  agile  mindset  supports  CMMI  model  intent, 
we  found  an  easy  implementation  that  we  cannot  work  without. 
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Thank  You! 

Other  Questions...? 


Part  of  the  content  is  a  result  of  contribution  from: 

Mr.  Elmohanned  Ahmed  (elmohanned  ahmed@ravac<  3m  ) 
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(CMMI®-ACQ)  Processes  to 
Improve  Organizational  Workforce 
and  System  Acquisition 
Performance 

Dr.  Kenneth  E.  Nidiffer  &  Rick  Barbour 
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Overview 


•  Is  your  organization  working  towards  achieving 
acquisition  excellence? 

-  The  application  of  systems-engineering  to  improve  the 
workforce  may  be  part  of  the  answer! 

•  What  are  the  rate-limiting  variables/drivers  that 
limit  success? 

•  How  can  the  CMMI  ®-  ACQ  model  be  used? 


Achieving  Acquisition  Excellence  via  Effective  Application  of  CMMI® -ACQ 
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An  Effective  Process  for  Major  Defense 
Systems  -  but  not  very  agile 
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-  Software  Engineering  Institute 


DOD  Software  Acquisition  Process  Improvement  Programs 

DoD  Major  Events  and  Leadership  Rotation 


Gap  Analysj 


Cohen  I 

Rumsfeld 

I  Gates 

01-24-1997  to  01-20-2001  I 

01-20-2001  to  12-17-2006 

I  12-17-2006  to  Present 

DDR&E  Imperatives 


i.  Accelerate  delivery  of  technical  capabilities  to  win 
the  current  fight. 


2.  Prepare  for  an  uncertain  future. 


3.  Reduce  the  cost,  acquisition  time  and  risk  of  our 
major  defense  acquisition  programs. 


4.  Develop  world  class  science,  technology, 

engineering,  and  mathematics  capabilities  for  the 
DoD  and  the  Nation. 


Source:  The  Honorable  Zachary  J.  Lemnios 
Director,  Defense  Research  and  Engineering 
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External  Forces:  Defense  Acquisition  Approach 

Systems  Engineering  is  key  discipline 


Source:  The  Honorable  Zachary  J.  Lemnios 
Director,  Defense  Research  and 
Engineering 


h* 


5  to  15  Years 
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Solution 
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&  Deployment 
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Development 

•CDR 

MS 
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LRIP  !  FRP 

1 

•  Op  needs 
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•SCR 

4- 

•AoA 

•  Red  Taming 

•  SEP 

•  T&E  Strategy 

▲ 

•  System  Spec 
•SEP 

•  T&E  plan 

•  SRR  &  PDR 

•  Prototype  dev 
•TRA 

▲ 

•TRR 

•  Dev  test  &  eval 

•  Initial  Ops  Test 

&  Eval 

•  PRR 

•TRA  Update 

▲ 

: 

•  Transition 

•  Ops  Test  &  Eval 

•  Training 

I 


Operations 

•  Sustainment 

•  Disposal 


ves 


Technology  and  Risk  Reduction 

•  Technology  “push”  investment 

•  Technology  maturation 

•  Phenomenology  measurements 


70-75%  of  Cost  Decisions  Made  Prior 
to  Milestone  A 

Impact  72%  of  Total  Life  Cycle  Costs 
=^0-  Software  Engineering  Institute 


AoA  -  Assessment  of  Alternat 
DP  -  Developmental  Planning 
MDD  -  Material  Development  Decision 
SCR  -  System  Concept  Review 
SRR  -  System  Requirements  Review 
SEP  -  System  Engineering  Plan 
PDR  -  Preliminary  Design  Review 
CDR  -  Critical  Design  Review 
TRR  -  Test  Readiness  Review 
PRR  -  Production  Readiness  Review 
LRIP  -  Low-Rate  Initial  Production 
FRP  -  Full  Rate  Production 
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23  Principal  Actions  to  Improve  Efficiency 
within  5  Major  Areas  (14  September  2010) 


1.  Target  Affordability  and  Control  Cost  Growth 

2.  Incentivize  Productivity  and  Innovation  in  Industry 

3.  Promote  Real  Competition 

4.  Improve  Tradecraft  in  Services  Acquisition 

5.  Reduce  Non-Productive  Processes  and 
Bureaucracy 
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Defense  Science  Board  Report  & 
Public  Law  111  (Section  804) 


DOD  acquisition  process  is  too  long 
and  too  cumbersome  to  fit  the  needs  of 
the  many  IT  systems  that  require 
continuous  changes  and  upgrades 


The  National  Defense  Authorization  Act 
for  Fiscal  Year  2010  (“the  Act”)  Public 
Law  111  includes  a  significant  set  of 
legislative  provisions  that  modify 
Department  of  Defense  (DoD) 
procurement  policies  and  practices. 


Task  Force  on 
Department  of 
Defense 
Policies  and 
Procedures  for 
the  Acquisition  of 
Information 
Technology 
March  2009 


The  National  Defense 
Authorization  Act  for  Fiscal 
Year  2010  (“the  Act”)  Public 
Law  111  (Section  804) 
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External  Forces 


AT&L  Civilians  -  Risk  of  Losing 


Years  of  Service 

— Traditional  Gone  ration  —Baby  Boomers  —Generation  X  —Generation  Y  —Millenium 

(Before  1946)  (1946-1964)  (1965-1976)  (1977-1989)  (1990  -  Present) 

Source:  DAU 


Bimodal  Demographics 
(Space  Industry) 


Area  of  Concern 

- A - , 


i  i 


Area  of  Concern 

- - * - - 


54%  of  the  S&T  Workforce  is 
Over  45  and  33%  wifi  be  retirement 
eligible  in  5  years 


Professional  Growth 
vs.  Time _ ^ 


Level 

4 


Source:  DAU 


Program  Systems  Engineer  (Master) 
(minimum  1 0  years  of  SE  experience) 


cR 


Level  3  -  Systems 
Engineer(Expert) 

(minimum  6  years  of 
SE  experience) 

Level  2  -  Acquisition  Engineer 
(Journeyman) 

(minimum  2  years  of  experience) 

Level  1  -  Acquisition  Engineer 
(Entry) 

(minimum  1  year  of  experience) 


55-59 

Source:  LMSC 


SPRDE/Systems  Engineering  Career  Field  SOUTCG'  DAU 
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Procurement  Budget 

vs.  DoD  Acquisition  Workforce 


DoD  Procurement  Budget  M  DoDAWF 


Increasing  #  of  Procurements  &  Complex  Systems  Coupled  With 
Huge  Decrease  In  Acquisition  Workforce 
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Recapture  Acquisition  Excellence: 

Revitalize  The  Acquisition  Workforce 

Problem 

•  Acquisition  capability  has  slowly  atrophied 

•  Organic  Workforce  reductions  -  23%  since  1999 

>  Force  shaping,  reduced  training,  retirements  of  critical  cost 
estimators,  price  analysts,  experienced  system  engineers, 
contracting  officers 

Initiatives 

•  Recapitalize  the  Acquisition  Corps/Training 

•  OSD  Funding  Increased  Numbers  and  Training  of  Organic 
Acquisition  Personnel 

It  May  Not  Be  All  About  the  Acquisition  Workforce  -  But 
Viable  Solutions  Must  Consider  the  Human  Element! 
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Project  Purpose 


Use  a  systems  engineering  approach  to  assess  acquisition 
training  and  organizational  training  processes  for 
improving  acquisition  excellence 


Experience 
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Business  Motivation 


Internal 

•  Improve  organization’s  performance  efficiencies  by  putting  in  place 
trained  workforce  that  can  leverage  suppliers’  capabilities  to  deliver 
quality  solutions  rapidly,  at  lower  costs,  and  with  appropriate 
technology 


External 

•  President  Barack  Obama  -  Mar/May  2009 
“The  government  will  assist  agencies  in  assessing  the  capacity  and 
ability  of  the  Federal  acquisition  workforce  to  develop,  manage,  and 
oversee  acquisitions”  and  Weapon  Systems  Acquisition  Reform  Act 
of  2009,  Public  Law  111-23,  22  May  2009 
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Summary  of  Systems  Engineering  Drivers 

External  Forces 

•  Increasing  size  of  untrained  defense  acquisition  workforce 

•  Retiring  of  experienced  and  capable  workforce 

Technological 

•  Accelerating  technological  changes  makes  systems  specific  acquisition 
training  difficult  at  best 

•  Identifying  future  competencies  to  ensure  most  relevant  training  content 
Human  Capital 

•  Changing  workforce  demographics  requiring  newer  methods  of  training 
and  management 

Client  Business  Environment 

•  Achieving  acquisition  excellence  in  a  fiscally  constrained  environment 
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What  Is  the  Cyber  Environment? 


Includes  all 

•  System  of 
Systems 

•  Architecture 

•  Services 

•  Netted 
Hardware/ 
Platforms 

•  People  who 
digitally  connect 
to  cyberspace 


-T  ■ 


Cyberspace  is  where  our  daily  work  and  our  priority 
missions  are  conducted 
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Federal  IT  Market  Growth 


In  the  next  five  years,  IT 

...spending  with 
contractors  will 
outpace  overall 
IT  growth 

billion  by  2015. 

--  Ben  Bain 


contractors  will  see  the  federal 
market  for  their  services  increase 
by  a  compound  annual  growth  rate 
of  5.4  percent  to  a  total  of  $1 1 1 .9 


Federal  Computer  Week 
April  8,  2010 
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KSLOC 


Increasing  Software  Lines  of  Code  & 
Complexity 


V*  UAVs 
y*  UCAV 

/F-35 


Aircraft  IOC,  Year 


10th  Annual  NDIA  CMMI  Technology 

Software  Engineering  Institute  Carnegie  Mellon  0^™.*  e.  wdi*.,,  nov.mb.r  2010  17 

©2010  Carnegie  Mellon  University 


Technology  Capability 


Technological:  Acceleration  of  Innovation  in  the 
21st  Century  -  Facilitating  Our  Ability  to  Build 
Move  Complex  Systems 
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Technological:  Moore's  Law  Holding  -  The  Number  of 
Transistors  That  Can  be  Placed  on  an  Integrated 
Circuit  is  Doubling  Approximately  Every  Two  Years 


Integrated  Circuit  Complexity 


Transistors 
Per  Die 
IO10-, 


104 


1965  Actual  Data 

MOS  Arrays  a  MOS  Logic  1975  Actual  Data 
1975  Projection 

Memory  16h 

Microprocessor 

256K 


256M 

128M 


512M 


"n  >  Itanium™ 

Pentium®  4 
^Pentiun#  III 
Pentium6  II 


8030 


4  -  3  ft  E  FiZ  t  S  •  € 

41' Vi  NpT  <4  *  m  -  m  -m 


I960  1965  1970  1975  1980  1965  1990  1995  2000  2005  2010 


■Source-:  I  ntei 


Technological:  Augustine’s  Law  Holding  -  Growth  of 
Software  is  an  Order  of  Magnitude  Every  10  Years 
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Technological:  Increasing  Rate  of  Adoption 

Electricity 
(1873) 


Automobile 

years 


Television 

=5r26) 


Telephone 
(1876) 
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Telephone  =  36  years 
Television  =  26  years 
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Source:  Rich  Kaplan,  Microso, . 
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Human  Capital:  Refocusing  University  Curriculums  - 
Alignment  of  Software  Systems  Engineering 
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OSD  Initiatives:  Graduate  Software  Engineering  Reference  Curriculum  (GSwERC) 
&  Body  of  Knowledge  and  Curriculum  to  Advance  Systems  Engineering  (BKCASE) 
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Human  Capital:  Changing  Demographics 

Demographics  of  workforce  are  changing  and  different  views  may 
emerge  with  four  generations  to  consider 

Generation  Y  professionals  entering  workforce  will  likely  necessitate 
non-traditional  training  techniques,  such  as  virtual  approaches 


Generation  X  Generation  Y/Millenials 

1965-1980  1981-2000 


Technically  advanced 
Prefers  informality 
Needs  structure  and  direction 
Direct/immediate  communicator 
Seeks  work/life  balance 


Technically  savvy 
Embraces  diversity 
Requires  supervision 
Indirect/virtual  communicator 
Demands  work/life  balance 


Silent  Generation 
1928-1945 


Hard  worker 
Respects  authority 
Work  is  obligation 
Formal  communicator 
Work/family  separation 


Baby  Boomers 
1946-1964 


Workaholic 
Questions  authority 
Works  efficiently 
Competitive 
Little  work/life  balance 
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Client  Business  Environment:  Increasingly  Complex 


Characteristics 


Market 


Industry 


Packaging 


Primary  Output 


Purpose 


Project  Duration 


Team  Size 


Ratio  of  Custom 
to  COTS/Reuse 


Agreement 


Customer 


#  Customers 


Focus 


Commercial 
Software  Products 


Commercial 


Software 


Products 


Software 


User  empowerment: 
effectiveness, 
efficiency,  creativity 


1-36  months 


1-1000's 


Software:  Low-high 


License 


External 


100's-l  ,000,000's 


Features,  Time-to- 
market,  Ship  it 


Information  Technology  & 
Internet  Financial  Services 


Information  technology  & 
internet 


Financial 


Services 


Integrated  system  engr  & 
HW&  SW  &  network 


Organization/business 

operations 


1-18  months 


1-1000's 


Business  logic:  High 
Others:  Low 


Service  level  agreement 


Internal  and  external 


1-1 ,000,000's 


User  experience.  Workflow 
cycletime.  Uptime 


Government  Aerospace 
Systems 


Government 


Aerospace 


Systems 


Integrated  system  engr  8 
HW&  SW  8  network 


Mission/science 

capabilities 


6  months  -  10  years 


1 0's-1 000's 


All:  High 


Contract 


External 


1 


Reliability,  Milestones, 
Interdependencies 


Source  -  Northrop  Grumman 
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Client  Business  Environment:  Acquisition  Shifts 


System 

Operation 


I 


“Acqi 

lisition 

1 

r 

2005  study  confirmed*: 

•  In  advanced  knowledge-based  organizations,  management’s 
desire  for  the  flow  of  knowledge  is  greater  than  the  desire  to 
control  boundaries 

•  Unlike  the  matrix  organization,  there  is  less  impact  on  the 
dynamics  of  formal  power  and  control 

Using  Communities  of  Practice  to  Drive  Organizational  Performance  and  Innovation,  2005,  APQ 
study 

Advanced  Knowledge-Based  Organizations  (Big  A) 


System 

Construction 


-Operational 


Constructive 


System 

Operation- 


System 
Conslmctjpn 


Program 

Management 


-Programmatic  \ 


Program 

Management 


From  “Science  and  Technology  to  Support  FORCEnev’  Raytheon  TD-06-008. 


“acquisition”  Used by permissi°n 


Ref:  Jim  Smith,  (703)  908-8221 , ids@sei.cmu.edu 
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Systems  Engineering  Approach 


Phase  1 

Identify/Collect  Data 


1 


Identify  Training 
Courses 


Identify 

Stakeholders 


Identify/Select 
Reference  Model 


Identify  Org. 
Training  Process 


Review  Legacy/ 
Current  Efforts 


Phase  2 

Perform  Gap  Analysis 

t 


0 

£ 

(/) 


0 

-o 

O 


0 

O 

c 

0 

0 

0 

a: 

O 

o 

< 


o 


Organizational 
Process  Gaps 


Framework 
Space  Gaps 


Selected  based  on 

•  amount/type  of  data  to  be  reviewed 

•  availability  of  a  reference  model 

•  requirements,  logical  and  physical  loops 

•  iteration  and  recurrision  activities 

•  access  to  key  stakeholders 


Training  Class 
Coverage  Gaps 


Phase  3 

Formulate/Codify  Findings 


1 


Findings,  Impacts, 
Recommendations 


Write  Draft  Report 


Phase  4 

Develop/Deliver  Results 


Write  Final  Report 


Communicate  Results  and 
Collect  Feedback 


HD 
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Project  Objectives 

During  assessment  Phase  1  project  objectives  were 
formulated  in  terms  of  five  questions 

•  Do  coverage  gaps  exist  in  the  training  of  acquisition  best 
practices? 

•  Do  gaps  exist  in  acquisition  training  on  the  unique  aspects 
of  the  client’s  system  acquisitions? 

•  Do  gaps  exist  in  the  training  of  the  client’s  acquisition 
lifecycle  framework  and  processes? 

•  Do  best-practice  gaps  exist  in  the  client’s  organizational 
training  processes? 

•  Do  gaps  exist  in  identifying  training  requirements  for 
satisfying  the  acquisition  workforce  core  competencies? 
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Assessment  Framework:  CMMI®-ACQ 


Operational  Need 


Focus  on  Acquisition  Best  Practices 

(Acquirer) 


10th  Annual  NDIA  CMMI  Technology 
Conference 

Dr.  Kenneth  E.  Nidiffer,  November  2010 
©2010  Carnegie  Mellon  University 


Software  Engineering  Institute 


Carnegie  Mellon 
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CMMI®  -ACQ  categories  and  process  areas 


Category 

Process  Area 

Acquisition 

Agreement  Management  (AM) 

Acquisition  Requirements  Development  (ARD) 

Acquisition  Technical  Management  (ATM) 

Acquisition  Validation  (AVAL) 

Acquisition  Verification  (AVER) 

Solicitation  and  Supplier  Agreement  Development  (SSAD) 

Process 

Management 

Organizational  Innovation  and  Deployment  (OID) 
Organizational  Process  Definition  (OPD) 

Organizational  Process  Focus  (OPF) 

Organizational  Process  Performance  (OPP) 

Organizational  Training  (OT) 

Project 

Management 

Integrated  Project  Management  (IPM) 

Project  Monitoring  and  Control  (PMC) 

Project  Planning  (PP) 

Quantitative  Project  Management  (QPM) 

Requirements  Management  (REQM) 

Risk  Management  (RSKM) 

Support 

Causal  Analysis  and  Resolution  (CAR) 

Configuration  Management  (CM) 

Decision  Analysis  and  Resolution  (DAR) 

Measurement  and  Analysis  (MA) 

Process  and  Product  Quality  Assurance  (PPQA) 

CMMI®  -ACQ  model  was 
developed  to  codify  best 
practices  to  help  organizations 
improve  acquisition  processes 

CMMI®  reference  models  have 
gained  significant  traction 
across  commercial  and  defense 
community  and  are  widely  used 
throughout  world  [CMMI 
Product  Team  07] 
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Summary  of  Results 

•  Strengths 

•  Areas  for  Improvement 

•  Lessons  Learned 
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Results  -  General  Overall  Strengths 

•  Excellent  coverage  in  the  training  of  acquisition  best  practices 

•  Adequate  number  and  variety  of  course  offerings 

•  Simple  but  adequate  training  facilities 

•  Consistency  of  course  material  &  presentation  layout  &  style 

•  Variety  of  media  used  for  announcing  upcoming  courses 

•  Scope  and  breath  of  Earned  Value  programs 

•  Knowledgeable  SME*  teach  classes 

•  Talented  instructor  workforce 

•  Intelligent  student  population 

•  Professionalism  of  the  training  staff 

•  Desire  to  improve 
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Representative  Results:  Question  1 


Question  1:  Do  Coverage  Gaps  Exist  in  the  Training  of  Acquisition  Best  Practices? 


Findings 

■  Detailed  findings  awaiting  client  approval 


Impacts 

■  Missing  opportunities  to 
~  attract  more  students 
~  provide  training  on  the  most  relevant  issues 
~  effectively  plan 
~  save  resources 

~  provide  a  richer  variety  of  courses 
~  continuously  improve  training  processes 


Recommendations  and  Considerations 

■  Conducting  a  review  to  assess  use  of  web-based 
and  non-traditional  acquisition  training 
Consider:  Leveraging  of  efforts  by  DAU, 
commercial  industry  and  academia 

■  Conducting  a  review  of  best  practices  for  training 
among  different  types  of  acquisitions 
Consider:  Developing  and  teaching  approaches 
that  focus  on  agile  and  SOA  acquisition 
approaches 

■  Making  a  better  use  of  repository  information 

Consider:  Using  DAU’s  Acquisition  Best 
Practices 

■  Putting  a  systematic  process  improvement  program 
in  place 

Consider:  Using  CMMI-ACQ  and  IDEAL 

■  Developing  a  strategic  plan 

Consider:  Socializing  plan  among  relevant 
stakeholders 
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Results 


Findings 

25 

Impacts 

20 

Recommendations 

23 

Considerations/  Potential  Solutions  - 
ways  to  address  some  of  the 
recommendations 

40 

Systematic  Improvement  in  Client’s  Organizational  Training  Processes  Needed 
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Lessons  Learned 


Tsunami 


•  Tsunami-like  impacts  on  new  acquisition 
training  requirements 

•  Rapid,  large-scale  disturbance  of 
current  training  needs  envisioned 

•  Forces  will  include  technological, 
human  capital,  external  and  government 
needs 

•  Training  departments  have  incorporated 
best  acquisition  practices  into  their  training 
courses;  however 

•  Mapping  of  core  competencies  to 
training  courses  needs  to  be  done 

•  Training  architectures  needed 

•  Developers  of  organizational  training 
processes  could  benefit  from  the  application 
of  systems  engineering 


Images  of  the  Ocean  Floor 
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Wrap  Up 


10th  Annual  NDIA  CMMI  Technology 

Software  Engineering  Institute  Carnegie  Mellon  0^™.*  e.  wdi*.,,  nov.mb.r  2010  35 

©2010  Carnegie  Mellon  University 


Contact  Information 

Dr.  Kenneth  E.  Nidiffer,  Director  of  Strategic  Plans  for  Government 
Programs 

Software  Engineering  Institute,  Carnegie  Mellon  University 
Office:  +  1  703-908-1117 
Fax:  +  1  703-908-9317 

Email:  nidiffer@sei.cmu.edu 
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Systems  Engineering  Drivers  for 
Improving  Acquisition  Excellence 

•  External  Forces 

•  Technological 

•  Human  Capital 

•  Client  Unique 
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Workforce  Size 
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Reference  Model 


Evaluated  client’s  acquisition  training  program  components 
using  Capability  Maturity  Model  Integration®  for  Acquisition 
(CM Ml®  -ACQ)  as  reference  model 
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Human  Capital:  Using  Core  Competencies 


+  Accurate  identification  of  required  competencies  are 
important  to  support  the  curriculum  review  and 
development  effort  needed  to  ensure  the  best  and  most 
relevant  training. 


■ 

Software  Engineer  III 

r 

Software  Engineer  II 

i 

Software  Engineer  1 

r 

Knowledge  Application  domain 

Procedural  design 
Cobol  &  Assembler 
Numerical  analysis 

Skills  Requirements  analysis 

System  design 
Project  management 
Debugging 

Process  Abilities  Integrated  team  design 
Fagan  inspections 
Test  procedures 
Change  control 


Competency  Family 
Software  Engineering 


Current  Resource  Profile  (initial  inventory) 


Workforce 

Staffing  by  Capacity  Lc 

jvel 

Competency 

1 

II 

III 

Software  Engineer 

17 

25 

12 

User  Training 

2 

8 

4 

Current  Resource  Needs  (one-year  cycle) 

Workforce 

Current  Staffing  Level  Nc 

ieded 

Competency 

i 

II 

III 

Software  Engineer 

23 

30 

15 

User  Training 

4 

9 

6 

Strategic  Workforce  Needs  (2-5  year) 

Workforce 

2010Staffing  Level  Nee 

ded 

Competency 

1 

II 

III 

Software  Engineer 

31 

35 

18 

User  Training 

4 

10 

8 
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NO  WARRANTY 


THIS  CARNEGIE  MELLON  UNIVERSITY  AND  SOFTWARE  ENGINEERING  INSTITUTE 
MATERIAL  IS  FURNISHED  ON  AN  “AS-IS"  BASIS.  CARNEGIE  MELLON  UNIVERSITY 
MAKES  NO  WARRANTIES  OF  ANY  KIND,  EITHER  EXPRESSED  OR  IMPLIED,  AS  TO 
ANY  MATTER  INCLUDING,  BUT  NOT  LIMITED  TO,  WARRANTY  OF  FITNESS  FOR 
PURPOSE  OR  MERCHANTABILITY,  EXCLUSIVITY,  OR  RESULTS  OBTAINED  FROM 
USE  OF  THE  MATERIAL.  CARNEGIE  MELLON  UNIVERSITY  DOES  NOT  MAKE  ANY 
WARRANTY  OF  ANY  KIND  WITH  RESPECT  TO  FREEDOM  FROM  PATENT, 
TRADEMARK,  OR  COPYRIGHT  INFRINGEMENT. 

Use  of  any  trademarks  in  this  presentation  is  not  intended  in  any  way  to  infringe  on  the 
rights  of  the  trademark  holder. 

This  presentation  may  be  reproduced  in  its  entirety,  without  modification,  and  freely 
distributed  in  written  or  electronic  form  without  requesting  formal  permission.  Permission 
is  required  for  any  other  use.  Requests  for  permission  should  be  directed  to  the  Software 
Engineering  Institute  at  permission@sei.cmu.edu. 

This  work  was  created  in  the  performance  of  Federal  Government  Contract  Number 
FA8721-05-C-0003  with  Carnegie  Mellon  University  for  the  operation  of  the  Software 
Engineering  Institute,  a  federally  funded  research  and  development  center.  The 
Government  of  the  United  States  has  a  royalty-free  government-purpose  license  to  use, 
duplicate,  or  disclose  the  work,  in  whole  or  in  part  and  in  any  manner,  and  to  have  or 
permit  others  to  do  so,  for  government  purposes  pursuant  to  the  copyright  license  under 
the  clause  at  252.227-7013. 
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Appraisals  and  CMMI  Gotchas 

Lessons  in  CMMI  Use 
and  Appraisal  Preparation 


Neil  Potter 
The  Process  Group 
help@processgroup.com 


Referenced  articles  are  at  www.processgroup.com/newsletter.htm 
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*  Introduction 

*  CMMI  Premise 

*  Documentation 

*  Configuration  Management 

*  Measurement  and  Analysis 

*  Supplier  Agreement  Management 

*  Project  Planning 

*  Project  Monitoring  and  Control 
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*  Integrated  Project  Management 

*  Training 

*  Equal-weighted  Process  Area  practices? 

*  Appraisal  Preparation  -  PIIDing 

*  Appraisal  Interview  Preparation 


©  Copyright  2007-2009  The  Process  Group.  All  rights  reserved. 


www. processgroup.com 


Version  1.4,  45min 


3 


THE 

PRC 

GRC 


CMMI  HAZARDS! 

Introduction 


Using  CMMI  or  preparing  for  an  appraisal? 

-  Avoid  the  hazard  of  creating  a  paper 
factory,  instead  focus  on  organizational 
results 

-  Avoid  putting  the  emphasis  on  the  less 
important  issues 

»  e.g.,  policy  recital,  training  records, 
emails  that  say  “We  assigned  this  to 
Fred” 

-  Spend  your  time  making  things  better,  not 
on  a  rote  exercise 

-  Know  some  common  blind  spots 
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CMMI  Premise 
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*  CMMI  practices  can: 

-  Reduce  project  risk 

-  Reduce  rework  and  costs 

-Improve  output  quality  and  predictability 

-Improve  productivity  through  process  improvement  and 
process  reuse 

•  CMMI: 

-Can  be  used  to  diagnose  current  state 

-  Provides  an  example  roadmap  forward 

»  Management/project,  engineering/organization, 
statistics/prediction,  variation/mean 
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Hazard:  Drowning  in  Documentation 


•  Easy  to  fall  into  the  trap  of  the  paper 
factory 


-  We  are  developers,  so  we  develop! 

-  What  we  really  need  is  guidance  for  our 
jobs 

»  Capture  best  organization 
engineering  and  management 
practices 

»  Not  necessarily  repeat  every  book 
known  to  mankind! 

•  What  problem  are  we  trying  to  solve? 


-  Make  engineering  easier,  quicker,  less 
hassle  -  NOT  MORE 


[Newsletter  “documentation”] 
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Configuration  Management  (CM) 

Hazard:  over-simplification 

*  CM  looks  pretty  straight  forward,  once  people  start  to  understand 
the  discipline 

*  Don’t  avoid  Configuration  audits  -  make  them  useful  [SP  3.2] 

-  Use  physical  audits  to  help  ensure  that  products  are  released 
correctly,  e.g., 

»  Verify  differences  between  source  and  release  =  change  list 
»  Compare  checksum  value  between  source  and  release 

*  What  problem(s)  are  we  trying  to  solve? 

-  Producing  the  right  stuff  and  getting  it  to  the  customer 

-  Keeping  track  of  our  stuff,  protecting  ourselves  from  loss 


SP  3.2:  Perform  configuration  audits  to  maintain  integrity  of  the  configuration  baselines. 
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Measurement  and  Analysis  (MA) 

Hazard:  skip  parts  or  overkill 


•  Organizations  often  have  metrics  but  entirely  skip  the  first  half  of  this 
Process  Area: 

-  Defining:  objectives,  metrics,  analysis,  reporting,  information  storage 

•  Or  take  the  other  extreme  and  overdo  measurement  and  goal 
definitions 

-  34  objectives,  a  procedure  for  documenting  objectives,  82  core  metrics 

•  Need  a  good  balance  for: 

-  Spending  enough  time  to  arrive  at  appropriate  goals 

-  Specifying  what  measures  are  needed 

-  Clarifying  how  they  will  be  analyzed  and  stored 

•  What  problem  are  we  trying  to  solve? 

-  Knowing  why  we  are  measuring  in  order  to  get  the  most  value  out  of  it 
and  not  waste  time  on  useless  metrics 


[Newsletter  “measurement”] 
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GP  2. 8/3.2  and  Over-simplified  MA 

Hazard:  I  measured  it  because  CMMI  SAID  I  HAD  TO! 


MA  comprises  of  only  7  PA  measures,  and  GP  2.8  and 
3.2  are  academic 

-  What  is  it  telling  you? 

•  What  problem  are  we  trying  to  solve? 

-  GP  2.8  (on  each  PA)  -  How’s  it  going  this  time? 

-  GP  3.2  (on  each  PA)  -  Are  the  PA  related  processes  as 
implemented  meeting  our  needs,  getting  better  or 
worse? 

-  MA  should  help  you  run  your  business,  not  just  CMMI! 


GP  2.8:  Monitor  and  control  the  <x>  process  against  the  plan  for  performing  the  process  and  take  appropriate  corrective  action. 

GP  3.2:  Collect  work  products,  measures,  measurement  results,  and  improvement  information  derived  from  planning  and 
performing  the  <x>  process  to  support  the  future  use  and  improvement  of  the  organization’s  processes  and  process  assets. 
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Project  Planning  (PP) 

Hazard:  skimping  on  size  estimation  and  risk  management 


Many  people  either  skip  size,  or  don’t  spend  enough  time  finding  a 
good  use  for  size  or  attribute  estimation  [SP  1.2] 


-  “My  project  size  is  2,000  hours” 

-  “I  estimate  Lines  of  Code,  but  track  effort” 

Others  underutilize  risk  at  the  project  level  [SP  2.2] 

-  Risks  should  come  from  the  team,  not  just  the  manager 

-  Risks  should  be  more  than  boilerplate  “We  might  not  have 
resources” 

-  Risks  should  be  made  very  visible  to  customers  +  management 
What  problem  are  we  trying  to  solve? 

-  Clarifying  how  big  the  project  is 

-  Understanding  what  can  really  go  wrong 

-  Thinking  through  potential  issues  ahead,  while  there  is  time  to 
react  /  recover 


SP  1.2  Establish  and  maintain  estimates  of  the  attributes  of  the  work  products  and  tasks. 
SP  2.2  Identify  and  analyze  project  risks. 


[Newsletter  “attributes”] 
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Project  Monitoring  and  Control  (PMC) 

Hazard:  missing  valuable  information  that  could  save  the  day 

•  No  useful  way  to  track  actual  work  progress  [SP  1.1],  e.g., 

-  Actual  work  effort  (labor) 

-  Actual  amount  of  work  accomplished  (size) 

•  What  problem  are  we  trying  to  solve? 

-  Use  data  to  determine  if  current  resource  expenditure  (hours  or 
money)  can  be  sustained 

-  Know  the  volume  of  work  and  how  much  each  project  actually 
costs 


»  How  much  we  lost  this  time,  or  how  much  future  projects  might  cost 

-  Proactively  manage  and  identify  re-planning  points  while  there  is 
time  to  recover 

»  Identifying  large  changes  in  effort  or  size 


SP  1 .1  Monitor  the  actual  values  of  the  project  planning  parameters  against  the  project  plan. 


[Newsletter  “attributes”] 
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Integrated  Project  Management  (IPM) 

Hazard:  not  having  proactive  visibility 

*  Not  use  thresholds  to  trigger  corrective  action  [SP  1 .5] 

-  At  Level  3,  corrective  action  and  escalation  are  more  objective  (“We 
are  10%  behind”)  than  emotional  (“I  think  things  will  speed  up”) 

-  Organizational  and  project  knowledge  are  used  to  establish 
thresholds 

*  Process  tailoring  not  based  on  organizational  learning  [SP  1.1] 

-  Level  3  is  often  interpreted  as  “Processes  are  standardized  across 
all  projects,”  rather  than  “Standard  processes  are  tailored  for  each 
project” 

*  What  problem  are  we  trying  to  solve? 

-  We  have  MEANINGFUL  data,  let’s  really  use  it! 

-  Have  organizational  wisdom  available  and  used 


SP  1 .5  Manage  the  project  using  the  project  plan,  the  other  plans  that  affect  the  project,  and  the  project’s  defined  process. 
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Integrated  Project  Management  (IPM) 
Without  Historical  Data? 

Hazard:  databases  full  of  data  are  not  enough! 

*  Organizational  Process  Definition  (OPD) 
and  IPM  not  well  understood 

-  OPD  sets  up  a  Process  Asset  Library  and 
measurement  repository  for  use  by 
projects  (IPM) 

-  Not  all  Lead  appraisers  know  or 
communicate  this 

*  What  problem  are  we  trying  to  solve? 

-  Run  projects  based  on  historical  and 
current  data 
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Do  Software  Engineers  Need  Training? 

Hazard:  trivial  training 


Project  Planning  (Sp  2.5) 

-  Make  sure  you  have  the  skills  for  THIS 
project 

Organizational  Training 

-  Make  sure  you  have  the  skills  for  current 
work,  and  work  to  come 

What  problem  are  we  trying  to  solve? 

-  Engineers  and  managers  don’t  have  the 
skills  to  perform  their  roles  correctly  (as 
per  process  definition)  and/or  efficiently 

-  Prevent  mistakes  due  to  lack  of  skills 
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Equal-weighted  Process  Area  practices? 

Hazard:  each  process  area  practice  is  treated  as  EQUAL 

•  Each  CMMI  practice  should  not  necessarily  be  equally 
weighted  during  implementation.  Example: 

-  Policy  vs.  estimating  effort  or  risk 
-Training  records  vs.  performing  validation 

•  The  correct  weighting  can  be  given  when  you: 

-  Focus  on  what  you  are  trying  to  accomplish  (real  jobs) 

-  Use  the  CMMI  and  its  components  to  improve 

-  Fix  real  problems 

•  What  problem  are  we  trying  to  solve? 

-  Real  world,  day-to-day  work  gets  better  (easier,  faster,  higher 
quality,  less  stress,  less  busy-work,  less  rework,  less  risk) 
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Appraisal  Preparation  -  PIIDing 

Hazard:  creating  documents  to  please  the  appraiser 

•  As  an  appraisal  date  approaches,  people  find  themselves 
focused  on  providing  required  appraisal  evidence: 

-  A  lot  of  time  can  be  wasted  chasing  down  documents 

-  When  practices  are  institutionalized  correctly,  the  evidence 
needed  already  exists 

*  What  problem  are  we  trying  to  solve? 

-  Evidence  should  never  be  created  to  please  an  appraiser 

-  Artifacts  examined  should  be  the  real  work  of  the  organization 

-  For  example,  evidence  of  responsibilities  could  be  an 
organization  chart  or  a  schedule  with  assignments 

*Practice  Implementation  Indicator 
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Appraisal  Interview  Preparation 

Hazard:  wasting  time  rehearsing 

•  Some  people  prepare  using  mock  interviews 

-  Appraisals  should  be  about  how  you  DO  YOUR  REAL 
work 

-  Interview  practice  might  make  folks  feel  more 
comfortable,  but  this  can: 

»  Induce  stress  over  remembering  to  say  the  right 
answers 

»  Focus  your  people  on  CMMI  terms  and  rote 
answers 

•  What  problem  are  we  trying  to  solve? 

-  Time  to  practice  for  an  appraisal  takes  away  from 
getting  real  work  done 

-  Participants  should  be  able  to  answer  the  questions 
because  the  answers  describe  how  they  do  their  jobs 
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Q& A 
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Additional  Slides 
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Supplier  Agreement  Management  (SAM) 

Hazard:  ill-advised  avoidance 

*  A  group  might  declare  SAM  Not  Applicable: 

-They  really  do  have  a  supplier,  but  are  used  to  dealing 
with  them 

*  Initially  there  are  no  suppliers 

-Then  suppliers  are  added,  but  SAM  is  not  invoked 

*  What  problem(s)  are  we  trying  to  solve? 

-Assessing  and  managing  risks  caused  by  suppliers 

-  Establishing  agreements  and  expectations  for  delivery 

-  Providing  visibility  into  supplier  activities  before  it  is  too 
late 
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Maturity  Level  4 

Hazard:  having  a  metric  or  statistics  wizard  is  enough 

•  Assume  that  if  we  can  just  find  that  one  magic  metric,  we  will 
be  Level  4  (maybe  even  5) 

-  It’s  not  really  about  a  metric  or  two;  it’s  about  using  statistical 
thinking  to  do  your  work! 

•  Assume  that  a  metrics  person  can  do  all  of  Quantitative 
Project  Management  (QPM) 

-  Allowing  project  managers  to  focus  on  their 

regular  day-to-day  tasks! 

•  What  problem  are  we  trying  to  solve? 

-  Understand  statistical  variation  and  remove  special  causes 

-  Run  projects  quantitatively  and  sub  processes  statistically 

-  Base  decisions  on  what  we  now  know  and  predict  ahead 
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Hazard:  numbers  alone  are  not  enough! 
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•  Very  specific  words  used  in  the  model 

-  Run  projects  quantitatively  and  sub 

processes  statistically 

»  Understand  statistical  variation 

»  Remove  special  causes  of 
variation 

»  Use  some  type  of  SPC 

•  What  problem  are  we  trying  to  solve? 

-  Make  business  decisions  based  on 
calculated  natural  bounds 

-  Use  data  to  predict  outcomes 
statistically 
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Code  Inspection  Defect  Density(with  trial  control  limits) 


Module# 


Manufacturing 
control  system 

00/C++ 

167  KLOC 

13  defects/KLOC 
in  code 

1  38 

defects/KLOC  in 
test 


[From  client  with 
permission] 


©  Copyright  2007-2009  The  Process  Group.  All  rights  reserved. 


www. processgroup.com 


Version  1.4,  45min 


23 


Maturity  Level  5 

Hazard:  not  building  on  statistically  stable  (L4)  processes 


Shifting 
the  mean 


Continual  improvement  means  measurably  improving  process 
capability  in  a  controlled  fashion. 


Chart  from  CMMI  Intro.  ©  2002  by  Carnegie  Mellon  University 
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Maturity  Level  5  (Cont.) 

Hazard:  not  building  on  statistically  stable  (L4)  processes 

*  It  is  easy  to  interpret  Level  5  Process  Areas  as  qualitative.  You 
might  think  that: 

-  Casual  Analysis  and  Resolution  (CAR)  could  consist  of 
brainstorming  causes 

-  Organizational  Innovation  and  Deployment  (OID)  could  be 
mistaken  for  qualitative  improvement 

»  Qualitative  improvement  is  L3  Organizational  Process  Focus 
(OPF)  and  Organizational  Process  Definition  (OPD) 

•  What  problem  are  we  trying  to  solve? 

-  Level  4  is  intended  to  collect  and  use  data  statistically  for 
prediction,  control  and  decisions.  Level  5  practices  build  on  that 
to: 

»  Reduce  variation  of  selected  sub  processes  (remove  common 
causes  of  variation),  AND  /  OR  shift  the  mean 
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Maturity  Level  4  and  5  Crack  Down? 

Hazard:  an  SEI  audit  takes  away  your  dreams  of  Level  4/5 

•  Some  appraisers  have  been  too  generous 

-  Did  they  NOT  understand  the  Model? 

-  Did  they  SELL  a  level? 

•  What  to  do  now? 

-  Re-educate  people  on  the  intent  and  details 
of  Level  4/5? 

-  Be  harsh  on  lead  appraisers  now? 

-  Take  away  levels? 

•  What  problem  are  we  trying  to  solve? 

-  Devaluation  of  Level  4  and  Level  5 

»  “I  have  a  vendor  in  <city  X>.  They  say 
they  are  Level  5  but  don’t  even  act 
Level  2.” 
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Buying  a  Level? 

Hazard:  doesn’t  help  run  your  business 

•  What  if  you  choose  “easy”  appraiser 

-  Has  your  business  improved? 

-  Giving  you  credit  for  too  much  can: 

»  Build  a  poor  foundation  for  the  future 

»  Upset  your  customer(s)  who  now  have 
higher  expectations  about  your  abilities 

»  Devalue  the  ratings 

»  Cause  more  audits 

•  What  problem  are  we  trying  to  solve? 

-  Someone  told  us  to  be  at  a  level,  so  we  are 
looking  for  the  quick  path 

-  CMMI  intent  is  to  set  you  on  an  improvement 
path,  not  to  pass  a  test 
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Raytheon 

Customer  Success  Is  Our  Mission 


Disruptive  Appraisals 

Prohibited 

Kelly  Lanier 
Maureen  Wampner 
Ralph  Williams 
November  2010 
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2005-2007  Appraisal  Metrics 


Raytheon 

Network  Centric  Systems 


Task 

2005-2007 

CMMI  Level 

5 

Regions 

5 

#  SCAMPI  Programs 

39 

#  Artifacts  in  Evidence 

10261 

#  Class  C  Appraisals 

14 

#  Class  B  appraisals 

4 

#  Document  Requests  During 

SCAMPI 

207 

#  SCAMPI  Findings 

8 

Length  of  SCAM  PI 

5  weeks 

Set  goal  to  reduce  cost  &  program  impact  while  sustaining  CMMI  Level  5 
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Baseline 
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■  NCS  Common  Process  Architecture  (CPA)  is  based  on  good  Engineering 
practices 

-  Not  built  around  passing  a  CMMI  appraisal 

■  Common  Process  Architecture  execution  is  in  place  and  remains  stable 

■  Improvement  priorities  are  based  on  strategic  NCS  business  goals 

-  Not  driven  by  CMMI  findings  or  CMMI  compliance 

-  Results  deployed  through  NCS  CPA 


Consistent  process  execution  minimizes  appraisal  preparation 
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Process  Ownership 


Raytheon 

Network  Centric  Systems 


■  Engineering  Disciplines  took  ownership  of  their  own  /processes 

-  Process  changes,  questions,  and  issues  go  back  to  the  owning  discipline 

-  Discipline  Directors  sponsored  organizational  improvement  projects 

■  Renioved  traditional_prgcess  group  “crutches”  to  encourage 
programs  accountability  for  process  execution 

-  Large  regional  process  groups  responsible  for  program  process  “stuff 

-  Measurement  teams  responsible  for  entering  and  analyzing  program  metrics 

-  Tailoring  teams  responsible  for  CPA  tailoring  and  program  work  instructions 
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Emphasis  on  Improving  Program 
Execution 
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■  Regional  Engineering  Process  Groups  replaced 
with  Engineering  Execution  Centers  focused 
on  improving  program  execution 

-  Provides  “bottoms-up”  improvement 

■  Improvement  projects  determined  by  NCS 
Business  Strategy  and  Engineering  Strategy 
elements 

-  Engineering  and  Technology  Strategy  Matrix 

-  Provides  “top-down”  systemic  improvements 

■  Process  Management  became  a  background 
task  with  a  small  set  of  resources 


Improvements  tied  to  Business  and  Engineering  goals 
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Strengthened 

the  Measurement  Program 
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■  Enabled.  Program  accountability  and.  ownership  of  their  own  metrics 

■  MEASUREthis! 


■  Leading  Indicators  Review  with  VPs 


■  Leading  Indicator  Deep  Dives 


Strong  measurement  program  integral  to  program  execution 
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Process  Sustainment  Monitoring 


Raytheon 

Network  Centric  Systems 


■  Interim  CPA  Checks  (iCPAs) 

-  Smaller,  still  frequent 

-  An  issue  in  one  region  is  an  issue  in  all  regions 

■  Identified  a  common,  smaller  set  of  documents  for  evidence 

-  Pareto  Principle 

■  Identified  and  resolved  issues  early 

-  Started  immediately  after  2007  SCAMPI-A 

■  Eliminated  backslide 

Achieving  Excellence  Together 
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Tailored  the  Appraisal  Process  and 
Mindset 
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■  Set  Appraisal  Team  expectations 

-  Some  level  of  discovery  will  be  required 

-  Expected  to  be  able  to  determine  if  CMMI  practices  were  implemented  by 
evidence  and  affirmations  provided  without  “extreme  annotation” 

■  Reduced  unique  evidence  artifacts  collected  and  reviewed  10,000+  to 
1,600 

-  Higher  number  of  artifacts  used  to  be  a  badge  of  honor  for  appraisal  teams 

■  Validated  evidence  with  internal  appraisal  team  members  rather  than 
using  full  appraisal  team 

■  Deleted  unnecessary  briefings  to  the  organization 

-  Opening  briefing,  Draft  Findings 

-  Eliminated  :90  per  program  participant 


Easier  for  programs  rather  than  easier  for  the  appraisal  team 
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■  Used  “show  and  tell”  format  to  allow  programs  to  better  explain  their  work 

-  DOORS 

-  Configuration  Management  Tools 

-  Active  Risk  Manager 

■  Executed  interviews  in  parallel  using  VTC 

■  Reduced  full  appraisal  team  review  of  appraisal  evidence  by  4  hours 

-  Accepted  Mini-Team  “FI”  recommendations 

■  Presented  Final  Findings  via  Netcast  to  allow  appraisal  participants  to 
participate  from  their  desks 
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Three  Year  Timeline 


Raytheon 

Network  Centric  Systems 


6/2007 
CMMI  L5 


11/2007 

Lead  Appraiser 
Support  for 
Reduced  CMMI 
costs 


1/2008-5/2009 
Interim 
CPA  checks 


7/2009 

Reduced  Evidence 
Approach 


5/2010 
CMMI  L5 


12/2007 
Solution  Area / 
Business  Goals 
Strategy  Matrix 


10/2008 
1st  of  22 
Measure  This! 
Classes 


10/2007 

EECs 

Established 


8/2009 
Appraisal 
Team 
Level  Set 
(Changing 
Mindset/Setting 
Expectations) 


1/2010 

Appraisal 

Process 

Tailoring 


Appraisal  accepted 
and  approved  by 
the  SEI  for 
public  use 
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Appraisal  Cost  Comparison 
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2005 

Class  C  1 

7/25  -  7/29 

2008 

iCPAI 

4/7-4/18 

-12% 

Class  C  2 

9/27-9/29 

iCPA2 

6/16-6/27 

Class  C  3 

10/10-10/12 

iCPA3 

7/28-8/8 

Class  C  4 

11/14-11/16 

iCPA4 

9/15-9/26 

Class  C  5 

12/6-12/8 

iCPA  5 

12/1-12/12 

Class  C  6 

12-Dec 

2006 

Class  C  7 

1/16-1/18 

2009 

iCPA6 

3/2 -3/6 

-59% 

Class  B 1 

2/6-2/17 

iCPA7 

3/10-3/12 

Class  C  8 

4/4-4/5 

iCPA8 

5/4-5/8 

Class  B  2 

4/24-5/5 

iCPA9 

7/13-7/17 

Class  B  3 

7/10-7/21 

Class  C  9 

8/14-8/16 

Class  B  4 

9/11-10/13 

iCPA9 

7/13-7/17 

2007 

Class  CIO 

1/3-1/5 

2010 

Class  B 

9/28-10/16 

-76% 

Class  C  1 1 

1/29-2/2 

Review 

3/10-3/11 

Class  C  12 

2/14-2/15 

SCAMPI 

5/3-5/14 

Class  C  13 

2/26-3/2 

Class  C 14 

4/2 -4/6 

Readiness 

Review 

4/30-5/1 

SCAMPI 

5/7 -6/1 

Achieved  62%  Reduction  in  Cost  in  2010 
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Measureable  Improvements 
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2007 

2010 

CMMI  Level 

5 

5 

Regions 

5 

5+Satellite 

#  SCAMPI  Programs 

39 

21 

#  Artifacts  in  Evidence 

10261 

1596 

#  Class  C  Appraisals 

14 

9 

#  Class  B  appraisals 

4 

1 

#  Document  Requests 
During  SCAMPI 

207 

21 

#  SCAMPI  Findings 

8 

0 

Length  of  SCAMPI 

5  weeks 

8.5  days 

84% 
Evidence 
Reduction 


75% 

Fewer 
Class  Bs 
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Reduction  in  Program  Impact 
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■  “The  appraisal  approach  you  guys  took  this  time  around  was  much  less  stressful 
on  our  program” 

■  “I  had  heard  that  being  an  appraisal  program  was  a  lot  of  work,  but  it  hasn’t  been 
a  big  deal  at  all” 

■  “I  didn’t  have  any  program  managers  yelling  at  me  about  the  appraisal  impact  this 
time  around” 


■  “I  didn’t  even  realize  we  were  having  an  appraisal” 


Total  #  Times  Hrs  Recorded  by  Program 


Approximately  how  many  hours  did  you  spend  on 
Collecting  evidence  for  the  appraisal? 


Quotes  and  survey  results  from  the  programs 
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Raytheon 

Network  Centric  Systems 


Congratulations 

NCS  ENGINEERING 

MM  (Level 


CMMI 


5 


May  2010 


Thank  you  for  your  commitment 
to  program  execution  excellence! 
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Agenda 

■  The  Past 

■  How  Did  We  Do  It? 

■  Results 

■  The  Future 

■  Summary 


Raytheon 

Network  Centric  Systems 


11/18/2010 
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The  Future 


Raytheon 

Network  Centric  Systems 


■  We  will  use  this  approach  for  all  future  appraisals 

■  Business  standards  of  program  execution  naturally  result  in 
success  without  employing  disruptive  efforts  to  prepare  for 
and  conduct  appraisals 


11/18/2010 


Page  1 9 


Agenda 

■  The  Past 

■  How  Did  We  Do  It? 

■  Results 

■  The  Future 

■  Summary 


Raytheon 

Network  Centric  Systems 
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Summary 


Raytheon 

Network  Centric  Systems 


■  CPA  execution  remained  stable  after  the  2007  SCAMPI  A 

■  Focused  on  improving  program  execution  rather  than  achieving  CMMI 
Level  5 

-  Emphasis  on  managing  with  metrics 

-  Discipline  and  program  ownership 

-  Process  not  separate  from  program  execution 

■  Partnered  with  our  lead  appraiser  to  tailor  the  appraisal  process  to  be  as 
minimally  invasive  and  cost  effective  as  possible 

■  Changed  appraisal  mindset  to  minimize  impact  on  programs 


Reduced  appraisal  costs  &  program  impact  while  sustaining  CMMI  Level  5 
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GENERAL  DYNAMICS 

Land  Systems 


Put  Your  Defect  Data  to  Work: 
Using  Process  Performance  Models  and  Root  Cause  &  Corrective  Action 


Margaret  Corr 
Dawn  Jaskolski 
David  Sobetski,  PMP 

November  17,  2010 


Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  Log  No.  2010-109,  dated  10-29-10 


Agenda 

•  GDLS  Overview 

•  Software  Quality  System  Overview 

•  Peer  Review  and  Software  Problem  Reporting  Tools 

•  Defect  Containment  Effectiveness  Model  (DCE) 

•  Process  Performance  Models  (PPM) 

•  Process  Performance  Baselines  (PPB) 

•  Root  Cause  and  Corrective  Action  (RCCA) 

•  Benefits  &  Challenges 
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Land  Systems  Products 


GENERAL  DYNAMICS 

l_0  fl  d  Syste  m  S  Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  Log  No.  2010-109,  dated  10-29-10 


Software  Quality  System  Overview 


PPM 


Ph«s»  Conumiwt  Etl»c*ivfn»ss  [PCE  =  Cowan  CfrfKts  I  To«  Otf+cts  *  tOOx] 


v:  ■  L  f.  ■.  i rr r  \- '  :v,l 


£/■  jMgjWjj 


Optimization 


PPB 

Monitoring 


RCCA 

Prevention 


PCE 

Identify 


✓ 


■X 
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Defect  Data 


Optimization 


Monitoring 


Phut  ConurnntM  EHtctimtss  IPCE « CoM«ntd  Otftctt  I  Toni  Otftcts '  100X] 


Dtltc*  CortMnfntnt  EHtcovtfiut  |OC£  ■  Phut  Otltctt  /  (Phut  Often  •  Pofl  Rtkut  Oftcnl ' 


Tottl  Cortuwtw  EHtctivtfttn  [TCE !  PitRtltut  Pttcn  I  Toni  Often '  MOxl 


IlDtitc^  thy  tKtot  >o  t  »vtloomW  c* 


■TOT 

Identify 


Prevention 


I  - 1 

• 

a  n 

Cxwm  (Winn  0or 

—  C>— ytrt 

flL 
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Defect  Data 

•  Defect  Origin 

•  Root  Cause 

•  Defect  Severity 

•  Peer  Review  Data 

•  Software  Problem  Reporting  Data 
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Defect  Origin  and  Root  Cause 


The  Root  Cause 

•  Selection  changes 
based  upon  the 
Defect  Origin 
selected. 

•  Chosen  based  on 
what  is  determined 
to  be  the  root  cause 
of  the  problem  and 
where  it  originated 
(Defect  Origin). 

•  Not  applicable  for 
editorial  issues. 


Issue  Category  (Defect  Origin) 

Root  Cause 

Planning 

Clarity 

Completeness 

Scope  Change  Per  Higher  Level  Documents 

Trace  to  Higher  Level  Documents 

Requirements 

Ambiguity 

Testability 

Traceability 

Design 

Architecture 

Detailed  Design 

Interfaces 

Traceability 

Code 

Data  Definition 

Data  Handling 

Initialization 

Logical  Error 

Standards 

Trace  to  Design 

Test 

Pass/Fail  Criteria 

Requirement  Traceability 

Test  Environment 

Scope 
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Defect  Severity 


•  It  is  important  to  categorize  the 
severity  of  the  issue  correctly.  This 
categorization  helps  the  project 
and  organization  identify  where 
process,  tool,  and  training 
improvements  are  needed. 

•  Major  and  Minor  issues,  Severity  1 
and  2  respectively,  are  considered 
technical  defects  that  generally 
require  further  analysis  and 
prevention  measures. 

•  Editorial  issues,  Severity  3,  are 
considered  non-technical  defect 
and  are  not  used  in  the  generation 
of  project  defect  prevention 
metrics. 


Defect  Origin: 

Planning  T 

Root  Cause: 

Clarjt^ - - - 

Issue  Severity:  / 

3 -Editorial  ^ 

,  em 

Location  in  Item:  ( 

[  Select  One  ] 

( — 

3  -  Editorial 

-) — 

1  Issue:  y 

2  -  Performance  Degraded  (Minor) 

/ph  l 

\l 

1  -  Mission  Critical  /  Safety  (Major)  S 

■r  j  '-  Al  1.1 

\  Jl 

Software  Defect  Severity  Decision  Tree  Diagram 


Potential  to  cause  Equipment  Failure 
or  Equipment  Damage,  Software 
Induced  Mission  Aborts,  Personnel 
Injury,  or  Personnel  Death 


Reduced  Performance,  Reduced  Reliability, 
Reduced  Safety,  Reduced  Effectiveness. 
Reduced  Maintainability 


An  editorial  or  cosmetic  issue  in  a  work 
product  that  could  have  been  caught  by  a 
spell  checker  that  is  not  visible  by  an  end 
user  or  that  suggested  change  in  phrasing  or 
verbiage. 

A  work  product's  performance  is  not 
degraded. 
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Peer  Review  Tool 


Root  Cause 
•  Dependent  on 
the  selected 
Defect  Origin 
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Software  Problem  Reporting  Tool 


Phase  Found  In 

This  is  the  lifecycle 
phase  that  the 
work  product  was 
in  when  the  defect 
was  identified  or 
found 


Defect  Origin 

This  is  the  lifecycle 
phase  that  the 
work  product  was 
in  when  the  defect 
was  injected  into  it 


Issue  Type: 

[STR  ^ 

Problem  Type: 

fsw  j*] 

Phase  Found  In: _ 

|  Independent  Software  Test  T  | 

Phase  Text: 

"REQUIRED  if  Phase  Found  In  =  "'Other  Fiekr. 


Date  Created: 

201 0-1 0-1 5  OS:  09:29 

Status: 

REVIEW/ANALYSIS 
Version  Identified  Ire 

pM 


i 

0 

Related  System 

Requirement: 

Test  Name: 

1 

Responsible  Engineer: 

Test  Engineer: 

[cORR.M. 

IcORR,  S. 

Defect  Of  ig  in:  Root  Cause 

|  Requirements  ^  |  [Select  One] 

^  Req  u  irements  Sta  nd  a  rd 

‘REQUIRED  if  Root  Cause 


Issue  Title: 


| Demonstration  CMMI-  San 
Items 


■0 

0 


Severity 

•  Safety/Mission 
Critical 

•  Performance 

Originator: 
C0RR,  Id. 

Degraded 

Severity: 

•  Performance  NOT 

|2.  Performance  Degraded 

Version  Injected  In: 

|  [Select  One] 

Degraded 

Regression  Problem: 

0  3S 


Deferred: 

[nJ  U 


Root  Cause 

Based  on  the 
defect  origin. 
Identifies  the  root 
cause  of  the  defect 
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Defect  Data  -  Benefits  and  Challenges 


•  Benefits: 

•  Challenges: 

71  Automation 

71  Data  Integrity 

■  Tools  enforce  a  more 

■  Consistently  identifying 

consistent  process 

defect  severities  between 

■  Simpler  data  collection 

technical  and  editorial 

■  Real-time  metrics 

■  Size  data  is  often 

7i  Consistent  data  between 

entered  incorrectly 

the  Peer  Review  and 

■  Root  cause  data  is  often 
inconsistent 

Problem  Reporting  Tools 

1  u 

7i  Guidelines  and  Help 

71  Identifying  definitions  for 
root  causes 

linked  into  the  tools  for 

easy  access 

71  Educating  employees  on 

definitions 
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Defect  Containment  Effectiveness 
(DCE)  Model 


Plus*  Found  In 

Ortgtr, 

R*ow«mnts 

Omw 

Cod*  Sottwt*T*st 

Post  SoHwu* 

Iwt  alien 

Total  0«4*cts 

Xhs*tttonR*t 

TCE 

R*JUHH7WMi 

181 

to 

43  B 

17 

26$ 

37X 

94X 

0*!iqn 

98 

5  0 

0 

#3 

HX 

BOX 

Cod* 

2001  B 

40 

255 

36X 

84X 

T»lt 

J_ II 

4 

91 

OX 

PCEs 

Phas*  Containrwnt  EH*«iv*n*ss  [PCE  =  Contamtd  0*f*«s  1  Total  D*f*cts  ‘  100x1  f 

DCE  = 

D*l*«  Containmwi  Efl*ctw*n*ss  [CCE  =  PKas*  0*f*cts  I  IPKas*  0*l*cts .  Post  FW*as*  0*i*cts| '  Bto^ 

TCE« 

Total  Ccntammw  Etf*ctni*n*ss  (TCE  «  Pt*R*l*as*  0*f*cts  1  Total  D*l*«ts  *  BOX]  aF 

Contain^  0*<*ys; 

Escape  D*f*cts  = 

DCE 

Identify 


|  - | 

• 

y\  □ 

— »'>T4K4' 

K" 

rtf 

/  /  /  x  / 


Prevention 
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DCE  Model  -  Introduction  and 
Definitions 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase  Found  In 

Origin 

Requirements 

Design 

Code 

Software  Test 

Post  Software 

Iteration 

Total  Defects 

*  Insertion  Rat* 

TCE  ; 

Requirements 

181 

10 

43 

15 

17 

266 

37* 

94* 

Design 

98 

5 

0 

0 

103 

14* 

100* 

Code 

200 

15 

40 

255 

36* 

34* 

Test 

37 

4 

91 

13* 

96* 

Total  Defects  715 


PCE  1 _ 63*1  95* 

73* 

96*1 

DCE 

60* 

34*| 

PCE  = 

1  Phase  Containment  Effectiveness  [PCE  =  Contained  Defects  /  Total  Defects '  100*] 

DCE  = 

Defect  Containment  Effectiveness  [DCE  =  Phase  Defects  /  (Phase  Defects  +  Post  Release  Defects) '  100*] 

TCE  = 

Total  Containment  Effectiveness  [TCE  =  PreRelease  Defects  /  Total  Defects '  100*] 

Contained  Defects  = 

1  Defects  that  are  found  during  the  phase  that  created  them. 

Escaped  Defects  =| 

1  Defects  that  escape  to  a  subsequent  development  or  delivery  phase. 

4 


‘THIS  IS  SAMPLE  DATA 


•  Phase  Containment 
Effectiveness  (PCE): 
tracks  the  ability  of  each 
phase  to  find  defects 
before  they  escape  that 
phase 

•  Defect  Containment 
Effectiveness  (DCE): 
tracks  the  ability  of  each 
phase  to  find  defects 
passed  to  it  by  upstream 
phases 

•  Total  Containment 
Effectiveness  (TCE): 
tracks  the  ability  of  the 
project  to  find  defects 
before  they  are  released 
(post-release) 


•  Source  data  is  Major  and  Minor  defects  collected  from  the  peer  review  database  and  problem  reporting  database 

•  Phase  Defects:  The  defects  that  were  found  in  a  specified  software  development  lifecycle  phase 

•  Post  Release  Defects:  The  defects  that  were  found  in  a  phase  outside  of  the  software  development  lifecycle  phase 

•  PreRelease  Defects:  Defects  that  were  found  in  a  software  development  lifecycle  phase  in  the  current  software  version 
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DCE  Model  -  Goals 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


1.  Early  Defect  Detection 


Phase  Found  In 

Origin 

Requirements 

Design 

Code 

Software  T est 

Post  Software 

Iteration 

A 

Requirements 

181 

10 

43 

15 

17 

Design 

88 

5 

0 

0 

♦ 

Code 

200 

15 

40 

* 

Test 

87 

4 

/ 

**  l— 

f 

^ otal  Defects 

PCE 

95--: 

78v; 

86:: 

DCE  V 

12^ 

60^ 

34^ 

T otal  Defects 


Insertion  Rett 


wirnm 

14V- 


2.  Defect  Prevention 


1  Phase  Containment  Effectiveness  [PCE  =  Contained  Defects  i  Total  Defects  '  100v;] 

DCE  = 

Defect  Containment  Effectiveness  [DCE  =  Phase  Defects  t  (Phase  Defects  Post  Release  Defects) '  100v;] 

i  TCE  = 

Total  Containment  Effectiveness  [TCE  =  PreRelease  Defects  t  Total  Defects  '  100v;] 

Contained  Defects  = 

1  Defects  that  are  found  during  the  phase  that  created  them. 

Escaped  Defects  =1 

iDefects  that  escape  to  a  subseauent  development  or  deliveru  phase.  1 

‘THIS  IS  SAMPLE  DATA 


Goals  of  the  Model: 

1.  Early  Defect  Detection 

•  Increase  the  number  of  defects  found  within  the  phase  they  originated  in 

>  Phase  Containment  Effectiveness  (PCE) 

•  Reduce  the  number  of  defects  escaping  to  later  development  phases 

>  Defect  Containment  Effectiveness  (DCE) 

2.  Defect  Prevention 

•  Reduce  total  number  of  defects  originating  within  a  specific  phase 

>  Total  Defects 

>  %  Insertion  Rate 


GENERAL  DYNAMIC! 
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DCE  Model  -  Usage 


When  Used: 

•  Data  collection  and  analysis  occurs  after  a  software  release 

Used  For: 

•  Identifying  organizational  process  and  technology  improvement 
needs 

•  Planning  of  subsequent  product  release(s) 

71  Consider  if  a  significant  number  of  defects  are  detected  in  any  particular  life 
cycle  phase 

71  Ensure  common  causes  of  defects  within  a  phase  or  phases  have  been 

identified  and  actions  to  further  investigate  causes  of  defects  are  assigned  and 
tracked  to  completion 

71  Understand  the  implications  of  not  addressing  the  common  causes  of  defects 

71  Ensure  implementation  of  preventive  corrective  actions  (e.g.  notify  developers 
of  common  defects,  increase  focus  on  suspect  or  problem  areas) 
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Origin  Phase 


DCE  Model  -  Analysis  Examples 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase  Found  In 

Origin 

Requirements 

Design 

Code 

Software  Test 

Post  Software 

Iteration 

Total  Defects 

*  Insertion  Rate 

TCE 

Requirements 

181 

10 

43 

15 

17 

266 

37* 

94* 

Design 

98 

5 

0 

0 

103 

14* 

100* 

Code 

300 

15 

40 

255 

36* 

84* 

Test 

87 

4 

91 

13* 

96* 

Total  Defects  715 


of  post 


Is  the  number 

release  defects  \ 

significantly  higher  ) 

than  expected? 


ted?  y 


PCE  1 _ 68*1  95* 

78* 

DCE 

60* 

PCE  = 

1  Phase  Containment  Effectiveness  [PCE :  Contained  Defects  /  Total  Defects "  100*] 

DCE: 

Defect  Containment  Effectiveness  [DCE :  Phase  Defects  /  [Phase  Defects  +  Post  Release  Defects] "  100*1 

TCE: 

1  Total  Containment  Effectiveness  [TCE :  PreRelease  Defects  /  Total  Defects '  100*] 

Contained  Defects : 

1  Defects  that  are  found  during  the  phase  that  created  them. 

Escaped  Defects  :| 

iDefects  that  escape  to  a  subsequent  development  or  deliveru  phase. 

‘THIS  IS  SAMPLE  DATA 


CWhat  is  the  phase  with 
the  lowest  PCE 
percentage?  Consider 
how  early  in  the  life 
cycle  this  phase  is. 


CIs  the  number  of  post 

release  defects  \ 

significantly  lower  J 

than  expected?  / 


How  does  the  PCE,  'v 
DCE  and  TCE  compare  \ 

to  the  last  release  or  ) 

other  projects? 


:s?  y 
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Defect  Containment  Effectiveness 
Model  Considerations  for  Analysis 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase 

Found  In 

Oriqin 

Requirement' 

Desiqn 

Code 

Software  Test 

Post  Software 
Iteration 

Total  Defects 

%  Insertion  Rati 

TCE 

Requirements 

181 

10 

43 

15 

17 

266 

37% 

94 54 

Desiqn 

38 

5 

0 

0 

103 

14% 

100% 

Code 

200 

15 

40 

255 

36  X 

84% 

Test 

87 

4 

81 

13% 

96% 

Total  Defects  715 


|PCE  I  68^1  35-/.I  78x|  96%^J 

loCE  ^  12x|  60x1 


PCE  = 

Phase  Containment  Effectiveness  [PCE  =  Contained  Defects  1  Total  Defects 1 100%1 

DCE  = 

Defect  Containment  Effectiveness  fDCE  =  Phase  Defects  1  (Phase  Defects  •  Post  Release  Defects)  •  100%] 

TCE  = 

Total  Containment  Effectiveness  fTCE  =  PreRelease  Defects  (Total  Defects  ‘  100%1 

Contained  Defects  = 

Defects  that  are  found  durina  the  phase  that  created  them. 

Escaped  Defects  = 

lOefects  that  escape  to  a  subsequent  development  or  deliverq  phase. 

*THIS  IS  SAMPLE  DATA 


I 


Software  Defect  Prevention 

Escaped  Defects  by  Origin 


Software  Requirements  Origin  Defects 


200 


Software Architecture'Design Origin  Defects 


°°a’ 


Phase  Found  In 


***** 


Phase  Found  In 


Software  Code  Origin  Defects 


SoftwareTestOrigin  Defects 


o 

g  150 


■STRS  ■=eerRevie# 


°-  JZ 1**~~*Z>*~ 


Phase  Found  In 


Phase  Found  In 


THIS  IS  S  AMPLE  D  AT  A 
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DCE  Model  -  Benefits  and  Challenges 


•  Benefits 

•  Challenges 

71  One  model  that  can  be 

71  The  model  can  be 

used  as  a  basis  of 

complex  to  understand, 

identifying  areas  for  both 

so  time  is  needed  to 

early  defect  detection  and 

educate  users 

defect  prevention 

71  Formula  variables  need  to 

7i  Provides  an  overall  view 

be  well  defined  and 

of  a  project’s  quality  with 

understood  before 

many  ways  to  slice  and 

generating  the  model 

dice  the  data 

71  Shows  the  macro  view  of 

71  Similar  project  data  can 

defects,  so  a  change  in 

be  rolled  up  as 

the  model  cannot  directly 

appropriate 

be  linked  to  a  specific 

improvement 

■  Not  measuring  a  sub¬ 

process 
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Process  Performance  Models  (PPM) 


PPM 

Optimization 


Monitoring 


Phut  ConurnntM  EHtctimtss  IPCE « CoM«ntd  Otftctt  I  Toni  Otftcts '  100X] 


Dtltc*  CortMnfntnt  EHtcovtfiut  |0C£  ■  Phut  Otltctt  /  (Phut  Often  •  Pofl  Rtkut  Oftcnl ' 


Tottl  Cortuwtflt  EHtctivtfttn  [TCE  =  PitRtltut  Otttcn  I  Toni  Often '  MOxl 


'  I  Often  thy  tKtPt  >0 f  *»>♦«**  dtvtlopmw  y  fthwi  fr«t 


■TOT 

Identify 


Prevention 


I  - 1 

• 

/  n 

K" 

flL 
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Origin  Phase 


PPM  -  Early  Defect  Detection 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase  Found  In 

Origin 

Requirements 

Design 

Code 

Software  Test 

Post  Software 

Iteration 

Total  Defects 

K  Insertion  Rate 

TCE 

Requirements 

131 

10 

43 

15 

17 

233 

37% 

94% 

Design 

OS 

5 

0 

0 

103 

14% 

100% 

Code 

200 

15 

40 

255 

3SK 

34^ 

Test 

37 

4 

31 

13% 

33k 

Total  Defects  715 


PCE  1 

l  SSkJx  95K 

78K 

33k 

□CE 

1  12K 

80K 

34K 

PCE  = 

Phase  Containment  Effectfeh^ess  [PCE  =  Contained  Defects  /  Total  Defects '  100K] 

□CE  = 

Defect  Containment  EffectiveneStDCE  =  Phase  Defects  /  (Phase  Defects  +  Post  Release  Defects] '  100K] 

TCE  = 

Total  Containment  Effectiveness  [TC^PreRelease  Defects  /  Total  Defects '  100K] 

Contained  Defects  = 

Defects  that  are  found  during  the  phase  thaNteated  them. 

Escaped  Defects  =[ 

1  Defects  that  escape  to  a  subsequent  developmelKor  delivery  phase. 

*THIS  IS  SAMPLE  DATA 


Peer  Review  Process  Performance 
Models  were  first  created  for 
Requirements  in  order  to  improve 
the  Requirements  PCE  through 
Early  Defect  Detection 
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Software  Metrics  Tool  -  PPM  Main  Page 


Software  Systems  Metrics 

Welcome,  David  M  Sobetski  3-  FAQ  s- 

Home  Projects  Manpower 

Resources 

Feedback  3-  Too 

Is  Portal 

Code  Selection  Criteria  Averages 

Peer  Review  Model 

With  Meeting 

Major/Minor  Defects  per  Review:  ll.llll 

Without  Meeting 

Major/Minor  Defects  per  Review:  ll.llll 

Actlon:  |  Prediction  _J 

Program:  pbrams 

Total  Peer  Review  Time:  ll  min 

Total  Peer  Review  Time:  ll  min 

|  Requirements 

Major/Minor  Defects  Found  per  Total  Peer 
Review  Time:  ll.ll  defects/hour 

Major/Minor  Defects  Found  per  Total  Peer 

Review  Time:  ll.ll  defects/hour 

Meeting:  pes\J 

Preparation  Recommendations  with  Meeting: 

Go  | 

s-  Number  of  Participants  Entering  Issues:  ll 

Families  of  models  based  upon: 

•  Prediction  /  Optimization  / 
Comparison 

•  Program  (New  development  & 
maintenance) 

•  Requirements  /  Design  /  Code 

•  Meetings  /  No  Meetings 


Requirements  Selection  Criteria  Averages 


With  Meeting 


Without  Meeting 


Major/Minor  Defects  per  100  Requirements  per  Major/Minor  Defects  per  100  Requirements  per 
Review:  ll. nil  Review:  ll.llll 


Total  Peer  Review  Time:  ll  min 


Total  Peer  Review  Time: 


Major/Minor  Defects  Found  per  100  Major/Minor  Defects  Found  per  100 

requirements  per  Total  Peer  Review  Time:  ll.ll  requirements  per  Total  Peer  Review  Time:  ll.ll 

defects/hour  defects/hour 

Preparation  Recommendations  with  Meeting: 


■  Average  Prep  Time  per  Participated:  ll  min 
Software  Requirements  Reviewed:  ll 


Abrams  Requirements  Selection  Criteria 
Averages 


With  Meeting 

Major/Minor  Defects  per  Review:  ll.ll 

Total  Peer  Review  Time:  ll  min 

Major/Minor  Defects  Found  per  Total 
Peer  Review  Time:  ll.ll  defects/hour 


Without  Meeting 

Major/Minor  Defects  per  Review:  ll.ll 

Total  Peer  Review  Time:  ll  min 

Major/Minor  Defects  Found  per  Total 
Peer  Review  Time:  ll.ll  defects/hour 


Preparation  Recommendations  with  Meeting: 


3-  Hold  Overviews  for  all  Peer  Reviews 
3- Average  Prep  Time  per  Invited:  ll  -  ll  min 
Number  of  Reviewers  Entering  Issues:  ll 
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PPM  Tool  -  Prediction 


Software  Systems  Metrics 

Welcome,  David  M  Sobetski  »  FAQ  »  Feedback  »  Tools  Portal 

Home  Projects  Manpower  Resources  Efficiency 

Overtime  Tools  Training 

Abrams  Requirements  with  Meeting  Prediction  Tool 

Estimated  Number  of  Reviewers  Invited:  pj - 


Estimated  Avg.  Prep  Time  per  Invited  (mins):  p[“ 
Estimated  Review  Meeting  Time  (mins):  j^“ 

Predict  | 


Predictions 


Number  of  Reviewers  Attended: 

n.n 

Number  of  Reviewers  Participated: 

n.n 

Number  of  Reviewers  Making  Comments: 

n.n 

Avg.  Prep  Time  per  Attended  (mins): 

n.n 

Avg.  Prep  Time  per  Participated  (mins): 

n.n 

Technical  Errors  [95%  Cl]: 

n.n  [+/-  n.n] 

Total  Peer  Review  Time  (mins)  [95%  Cl]: 

n.n  [+/-  n.n] 

User  can  enter  the  tool  on  their 
own  or  is  automatically  routed 
here  at  the  start  of  a  peer  review. 

User  can  predict  peer  review 
results  based  upon  number  of 
reviewers  invited,  preparation  time 
spent  by  the  reviewers,  and 
meeting  time. 
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PPM  Tool  -  Optimize  Time 


Software  Systems  Metrics 

Welcome^  David  M  Sobe 

:tski  s-  FAQ  s-  Feedback  s-  Tools  Portal 

Home  Proje 

:cts  Manpower  Resources  Efficiency  Overtime  Tools  Trainim 

3 _ 

Abrams  Requirements  with  Meeting  Optimization  Model 


Minimum 

Maximum 

Number  Reviewers  Invited: 

ij_j 

F — 

Avg.  Prep  Time  per  Invited  (mins): 

K 

F — 

Actual  Review  Meeting  Time  (mins): 

K 

F — 

Target  Errors  Found: 

FT~ 

Target  Total  Time  Spent  (mins): 

Goal: 

Maximize  Errors  Found 

n 


b)  of 


Optimize 


Optimized  Outcome 


Recommended  Number  of  Reviewers  Invited: 

n 

Expected  Number  of  Reviewers  Attended: 

n 

Expected  Number  of  Reviewers  Participated: 

n.n 

Expected  Number  of  Reviewers  Making  Comments: 

n.n 

Recommended  Avg.  Prep  Time  per  Invited  (mins): 

n.n 

Expected  Avg.  Prep  Time  per  Attended  (mins): 

n.n 

Expected  Avg.  Prep  Time  per  Participated  (mins): 

n.n 

Recommended  Actual  Review  Meeting  Time  (mins): 

n 

Expected  Technical  Errors  [95%  Cl]: 

n.n  [+/-  n.n] 

Expected  Total  Peer  Review  Time  (mins)  [95%  Cl]: 

n.n  [+/-  n.n] 
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PPM  -  Benefits  and  Challenges 


•  Benefits 

71  Optimize  resource 
utilization 

71  Better  able  to  predict  future 
project  performance 

71  PPM  improvements 
returned  as  high  as  400% 
increase  in  technical 
defect  detection 

71  Exceeded  our  defect 
Phase  Containment 
Effectiveness  (PCE)  goal 
of  90%. 


•  Challenges 

71  PPM  Model  usage 

■  Making  it  user-friendly  and 
easily  accessible 

■  Documenting  usage 

71  Stakeholder  buy-in 

71  Keeping  it  from  being 
personal,  i.e.  measuring 
the  process  &  product  and 
not  the  person  performing 
the  work 

71  Ensuring  data  integrity  in 
the  models 
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Process  Performance  Baselines  (PPB) 


Optimization 


PPB 

Monitoring 


Phut  ConurnntM  EHtctimtss  IPCE « CoM«ntd  Otftctt  I  Toni  Otftcts '  100X] 


Dtltc*  CortMnfntnt  EHtcovtfiut  |0C£  ■  Phut  Otltctt  /  (Phut  Often  •  Pofl  Rtkut  Oftcnl ' 


Toni  Coonnmtflt  EllKtivtotis  [TCE » PttRtltut  Otttcn  I  Toni  Often '  MOxl 


W^iht.tKtPtiOt^t^OtvtloomwyOtlrmt^t 


Identify 
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Process  Performance  Baselines  (PPBs)  -  Using  PPM 
Tool  Post  Peer  Review  Comparison  Analysis 


Software  Systems  Metrics 

Home  Projects  Mane 

lower  Resources 

Efficiency 

Abrams  Requirements  with  Meeting  Comparison  Tool 

Actual  Var  (%) 

Review  Meeting  Time  (mins):  p -  p - 

Avg.  Prep  Time  per  Invited  (mins):  p — —  p - 

Number  Reviewers  Invited:  p -  p - 

Number  Reviewers  Making  Comments:  p - “|  p - 

Technical  Errors:  rpj - 

Total  Peer  Review  Time  (mins):  p - 

Compare 


Comparison 


Predicted 

Actual 

Technical  Errors  [95%  Cl]: 

n.n  [+/-  n.n] 

n 

Total  Peer  Review  Time  (mins)  [95%  Cl]: 

n.n  [+/-  n.n] 

n 

1.  Based  on  the  comparison  data  was  this  peer  review  effective?  |  Select  One  ^  I 

2.  Will  a  re-review  be  held?  |  Select  One  A 

3.  Describe  decisions  or  analysis  made  as  a  result  of  using  the  Comparison  Tool. 

- 3 

_ I 

Submit 


Data  is  automatically  provided  to  the 
PPBs  when  the  peer  review  is 
completed. 

Users  provide  analysis  information  on 
their  results.  Special  causes  of 
variation  can  be  identified  and 
explained  at  the  time  the  data  was 
created. 
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PPBs  -  Automated  Metrics 


Peer  Review  Process  Performance  Baseline  -  0  ■  SI  *  ®  @  ■  & 

Project;  fpToject  ABC  v3~Software  P  myDefault 


Product:  (Software  Requirements  Specification 


t  |  Type:  |  Preparation  (Meeting)  ^  j 


ft^Graph~] 


Peer  Review  (Ordered  by  Date) 


Show 

Check /Uncheck  Aii  p 

$7  Goal 

$7  Avg  Tech  Issues  (X) 
Deviation 

f7  Lower  Control  Limit 
Upper  Control  Limit 
f7  Sigma  Lines  (cr) 
p'  Technical  Issues 


Legend 

O  Goal 

Avg  Tech  Issues  (X) 

■  Deviation 

■  Lower  Control  Limit 

■  Upper  Control  Limit 
■l  Sigma  Lines  (cr) 

■  Technical  Issues 


Statistical  Process 
Control  charts  are 
generated  live 
from  Peer  Review 
Data 

User  selects 

•  Project 

•  Work  product  type 

•  Meeting  information 

•  Technical  Defects  or 
Cost  (hours) 

•  Properties  to  display 
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PPBs  -  Analysis 


Peer  Review  Process  Performance  Baseline  ■  #  ■  %  -  §8  -•  @  -  /±> 

Project;  fpToject  ABC  v3~Software  P  myDefault 


Product:  (Software  Requirements  Specification 


t  |  Type:  |  Preparation  (Meeting)  ^  j 


j^GraphJj 


Peer  Review  (Ordered  by  Date) 


Show 

Check/Uncheck  Aii  $7 

17  Goal 

$7  Avg  Tech  Issues  (X) 
$7  Deviation 
$7  Lower  Control  Limit 
Upper  Control  Limit 
J7  Sigma  Lines  (cr) 
p'  Technical  Issues 


Legend 

O  Goal 


Avg  Tech  Issues  (X) 
Deviation 

Lower  Control  Limit 
Upper  Control  Limit 
Sigma  Lines  (cr) 
Technical  Issues 


Analysis 

•  Look  for  special 
cause  variation 

•  Monitor  for  trends, 
shifts  or  drifts  that 
may  highlight 
process  changes 

•  Determine  if  re¬ 
review  should  be 
performed 
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PPBs  -  Benefits  and  Challenges 


•  Benefits 

71  Peer  reviews  automatically 
feed  PPBs 

71  Control  charts  are 
automated  to  generate 
from  live  peer  review  data 

71  Process  is  monitored  to 
ensure  process  is 
performing  as  expected 

71  Quickly  deal  with  process 
deviations 

71  Special  causes  of  variation 
can  be  identified  and 
documented  as  they  occur 


•  Challenges 

71  Educating  employees  on 
the  use  of  the  PPBs 

71  Determining  when  to  set 
new  process  baselines 

71  Determining  where  to 
segregate  the  data 

71  Ensuring  data  integrity 

71  Upkeep  of  statistical  data 
(UCL,  Mean,  Std  Dev) 
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Root  Cause  &  Corrective  Actions 


PPB 

Monitoring 


Phut  ConurnntM  Eftotimtss  IPCE « Cortantd  Ottots  I  Total  Otto's  *  100X] 


Dtto'  CofiiirtTitw  Etto'ivtfitss  [DCE  =  Phut  Dttois  I  (Phast  Dtto's  •  Post  Rtlttst  Otto>s|  ‘ 


Tottl  Cotttnruw  Elto'wtfttss  fTCE ;  PrtRtltut  Otto's  ITottl  Otto's '  WOxl 


.lotto's  **  'Q  i  s^stw*  dtvtiopmto  » frtst 


RCCA 

Prevention 


ppm 


Optimization 


■TOT 

Identify 


JXtii-ftFiLf  With  Opfifrftlzatton  HMm 


✓ 


■X 
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Origin  Phase 


Root  Cause  and  Corrective  Action  (RCCA)  - 
Defect  Prevention  Using  the  DCE  Model 

•  Considerations  for  analysis 
71  Common  Cause  Defects 
71  Critical  Defects 

Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase  Found  In 

Origin 

Requirements 

Design 

Code 

Software  Test 

Post  Software 

Iteration 

Total  Defects 

*  Insertion  Rat* 

TCE 

Requirements 

101 

10 

43 

15 

17 

260 

37* 

94* 

Design 

00 

5 

0 

0 

103 

14* 

100* 

Code 

200 

15 

40 

255 

36* 

04* 

Test 

07 

4 

91 

13* 

96* 

Total  Defects  715 


PCE 

60* 

95* 

70* 

96* 

DCE 

12* 

60* 

34* 

PCE  = 

Phase  Containment  Effectiveness  [PCE  =  Contained  Defects  /  Total  Defects '  100*] 

DCE  = 

Defect  Containment  Effectiveness  [DCE  =  Phase  Defects  /  [Phase  Defects  +  Post  Release  Defects] '  100*] 

TCE  = 

Total  Containment  Effectiveness  [TCE  =  PreRelease  Defects  /  Total  Defects '  100*] 

Contained  Defects: 

Defects  that  are  found  during  the  phase  that  created  them. 

Escaped  Defects  =| 

[Defects  that  escape  to  a  subsequent  development  or  delivery  phase.  ] 

*THIS  IS  SAMPLE  DATA 
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Root  Cause  and  Corrective  Action  (RCCA)  - 
Steps  For  Performing  RCCA 

1.  Identify  a  Problem 

2.  Defect  Root  Cause  Analysis 

3.  Fishbone  Analysis 

4.  Identify  Corrective  Actions 

5.  Verify  Corrective  Actions 
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Root  Cause  and  Corrective  Action  (RCCA) 
Stepl:  Identify  a  Problem  -  Example 

Software  Defect  Prevention 
Containment  Effectiveness  Model 
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Root  Cause  and  Corrective  Action  (RCCA)  - 
Step  2:  Defect  Root  Cause  Analysis  -  Example 


Software  Defect  Prevention 

Escaped  Defects  by  Origin 


Need  to  identify  the  Root  Cause  of  why  the 
defect  escaped. 

-The  Code  Defect  Origin  by  Root  Cause  chart 
shows  that  the  root  cause  of  the  Post 
Software  Iteration  Phase  defects  was  Logical 
Error. 


1  Rinieh  H»E  Cmj*4 
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Root  Cause  and  Corrective  Action  - 
Step  3:  Fishbone  Analysis  -  Example 


THIS  IS  SAMPLE  DATA 
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Root  Cause  and  Corrective  Action  - 

Step  4:  Identify  Corrective  Actions  -  Example 


Root  Cause: 

(*usually  High  Probability) 

Corrective  Action: 

Too  many  new  developers  and  not 

Add  a  review  of  the  resource 

enough  experts 

allocation  plan 

Rushing  through  coding  to  meet 

Need  to  plan  more  time  for  peer 

deadlines 

reviews 

Incomplete  requirements 

Pilot  agile  development  methods 

Lack  of  static  analysis  tool(s) 

Allocate  effort  and  budget  to 
identifying  and  purchase  static 
analysis  tool(s) 

‘THIS  IS  SAMPLE  DATA 
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Root  Cause  and  Corrective  Action  - 
Step  5:  Verify  Corrective  Actions  -  Example 


Before: 


After: 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Software  Defect  Prevention 
Containment  Effectiveness  Model 


Phase 

Found  In 

Driqin 

Requirements 

Design 

Code 

Software  Test 

Post  Software 
Iteration 

Total  Defects 

%  Insertion  Rate 

TCE 

Requirements 

181 

10 

43 

15 

17 

266 

37% 

94% 

Desiqn 

Code 

IH- 55 

200 

15 

255 

36% 

Test 

87 

4 

Total  Defects  715 


Phase  Found  In 

Oriqin 

Requirements 

Desiqn 

Code 

Software  Test 

Post  Software 

Iteration 

Total  Defects 

%  Insertion  Rate 

TCE 

Requirements 

181 

10 

43 

15 

17 

266 

38% 

|94% 

Code 

200 

15 

235 

134%  | 

Test 

87 

4 

hkv; 

Total  Defects  695 


PCE 

DCE 


PCE 

DCE 


PCE  = 

Phase  Containment  Effectiveness  [PCE :  Contained  Defects  1  Total  Defects  *  100%] 

DCE  = 

Defect  Containment  Effectiveness  [DCE :  Phase  Defects  1  (Phase  Defects  ♦  Post  Release  Defects) '  100%] 

TCE: 

Total  Containment  Effectiveness  [TCE :  PreRelease  Defects  1  Total  Defects '  100%] 

Contained  Defects : 

Defects  that  are  found  durinq  the  phase  that  created  them. 

Escaped  Defects : 

1  Defects  that  escape  to  a  subsequent  development  or  delivery  phase. 

PCE  = 

Phase  Containment  Effectiveness  [PCE :  Contained  Defects  /  Total  Defects  ’  100%] 

DCE  = 

Defect  Containment  Effectiveness  [DCE :  Phase  Defects  1  [Phase  Defects  ♦  Post  Release  Defects]  ’  100%] 

TCE  = 

Total  Containment  Effectiveness  [TCE :  PreRelease  Defects  /  Total  Defects '  100%] 

Contained  Defects : 

Defects  that  are  found  durinq  the  phase  that  created  them. 

Escaped  Defects : 

Defects  that  escape  to  a  subsequent  development  or  delivery  phase. 

*THIS  IS  SAMPLE  DATA 


’THIS  IS  SAMPLE  DATA 


•  This  model  alone  cannot  be  used  to 
verify  the  results  of  the  corrective 
action  implemented. 

•  PPBs  need  to  be  created/modified  at 
the  sub-process  level 


GENERAL.  DYNAMICS 

l_0  fl  d  Syste  m  S  Approved  for  Public  Release,  Distribution  Unlimited,  GDLS  approved,  Log  No.  2010-109,  dated  10-29-10 


Root  Cause  &  Corrective  Actions  - 


Benefits  and  Challenges 


•  Benefits 

71  Able  to  eliminate  defects 
and  make  process 
improvements  to  prevent 
reoccurrence 

71  DCE  Model  showing  fewer 
total  defects  and  lower 
insertion  rates 


•  Challenges 

71  Focusing  on  prevention 
versus  a  correction 

71  New  methods  require 
training 

71  Identifying  where  the  best 
return  on  investment  is 

71  Maintaining  the  scope  of 
the  effort 

71  Ensuring  involvement  of 
SMEs 
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Challenges 


•  Stakeholder  buy-in 

71  Understanding  the  intent  and  purpose  of  the  quality 
system 

■  Getting  past,  “Why  do  I  have  to  do  this?” 

71  Keeping  it  from  being  personal,  i.e.  measuring  the 
processes  &  products  and  not  the  person  performing 
the  work 

•  Data  Integrity 

71  Trusting  the  data 

•  Documenting  savings  /  cost  benefit 
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What  Worked  Well 


•  Automation  -  keep  it  simple 

71  Web-based  tools  (easy  access) 

71  Integrated  tools 

71  Automation  leads  to  data  and  process  consistency 
71  Users  able  to  focus  on  products,  not  process 

•  Communication  and  education 

71  Strong  and  repeated  communication  with  users 

71  Educating  key  stakeholders  on  the  quality  system  and  its 
benefits  to  help  others  buy-in 

71  Updating  processes  and  guidelines  to  identify  usage  of 
quality  system 

7i  Incorporating  user  feedback 
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Contact  Information 


Margaret  Corr 

Software  Engineering  Process  Group  Lead 

Section  Manager,  Software  Process,  Tools  &  Environment 

38500  Mound  Road 

Sterling  Heights,  Ml  48310 

586-825-5787 

corrm@qdls.com 

Dawn  Jaskolski 

Software  Engineering  Process  Group  Lead 

Senior  Software  Engineer 

38500  Mound  Road 

Sterling  Heights,  Ml  48310 

586-825-4418 

iaskolsk@qdls.com 

David  Sobetski,  PMP 

Sr.  Specialist  Business  Processes,  Systems  Engineering  Process  Excellence 

38500  Mound  Road 

Sterling  Heights,  Ml  48310 

586-825-5362 

sobetski@qdls.com 
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Using  the  SEI  Models 

and 


Practices  to  Assure 


System  Lifecycle 
Deliverables  Quality  and 
Compatibility 
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The  Theory  in  the  Models  is  Nice 

However 

Real  Life  is  More  Complicated 

Much  More 
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*  Considerations  for  Delivery 

Management 

The  product  is  successful  when  the  cost  of 
development  and  manufacturing  will  drop  and 
your  profit  will  increase 

Produce  high-quality  (?)  products  within  shorter 
time  lines 

To  Correct  balance  between  time  and  cost  versus 
yield  and  quality  is  essential  to  maximize  return 
on  investment 


Considerations  for  Delivery 

Management 


|  Demonstration  of  the  scalability 

1  Partial  selection  of  what  to  optimize 

£;  •  Material 

•  Cost  of  product 

•  Design  for 

Scalability 

Availability 

Reliability 

Serviceability 

Maintainability 

Supportability 

Stability 

Reusability 

•  Sustainability  of  the  Technology  as  a  solution 


Main  Roadblocks  in  Delivery 

Management 


During  our  analysis  and  planning,  we  were  able  to  identify 
improvement  targets  in  main  lifecycle  areas  such  as 

•  operations, 
information, 

•  governance, 
people 

organizational  structure, 
portfolios, 

•  project  execution, 
finance. 

And  as  in  core  process  that  are  critical  to  the  system  success  such  as 
stakeholder  management,  technical  interfaces  and  integration. 


Military  Combat  Services  Support 
Challenges  in  the  C4ISR  Systems 
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Extended  Systems 


D  Major  Risk? 


A  Complex  Effects-based  Environment 


Integrated  Defense  Acquisition,  Technology,  and  Logistics  Life  Cycle  Management  System 
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Typical  Lifecycle  Description 
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Overall 

Management  View 


-i 


Simplified  View 


Project 

Budget 


SOURCE:  Smith  and  Reinertsen;  Developing  Products  in  Half  the  Time 


The  Priority  Balance 


Development 

Speed 


Product 

Cost 


Product 

Performance 


The  Compliance  Requirements 
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Design  Validation 


The  Challenge  Statement 

Organizations  that  need  to  manage  matrix  /  complex  business 
unit  with  functional  teams  or  systems  and  to  establish 
business  relationships  with  other  businesses  face  major 
challenges  including: 

The  need  for  creating  a  win-win-situation 

The  effort  to  align  business  processes  and  link  up  information 
systems  across  company  borders 

Organizations  do  not  know  how  to  efficiently  use 

interoperability  from  the  business  perspective  to  identify 
the  fundamental  artifacts  that  are  related  to  the  business 
interoperability 


Integrated  Risk  Management  Approach 


Common  Failures  -  1 


ganizational  risk  events  are  predominantly 
managerial,  not  technical. 

Lack  of  defining  business  objectives  in  quantitative  terms  and 
structure 

Inadequate  definition  of  ’Good  Enough’  level 

Inability  to  differentiate  different  business  objectives  and 
success  factors  for  the  different  domains  and  lifecycle  phases 

Inadequate  resource  usage  and  adjustment  to  Plan  and 
Objectives 

Failure  to  identify  and  manage  risks 
Poor  or  mismanaged  service  /  operational  requirements 
Uncontrolled  baselines,  no  configuration  management 
Misunderstood  business  /  operational  needs  and  objectives 


Common  Failures  -  2 


Poor  contractor  acquisition  or  management 
Lack  of  skills,  capability  and  training 

Poor  planning  and  tracking 

•  Value  Stream 

•  Equipment 

•  Resources 

•  Finance 

Poor  /  misuse  of  data  and  measurements 
Inability  to  estimate  accurately 
No  quality  assurance  /  control 
Poor  communications 


The  Operational  Need 


Management  capability  level  from  both  professional  and 
knowledge  level 

Performance  and  reporting  norms 

Self  management  and  self  discipline  maintaining  personal 
professional  and  knowledge  capabilities 

Individual  and  team  discipline 

Cooperation  and  knowledge  and  resource  sharing 

Appropriate  visibility  of  information,  data  and  capabilities 

Quality  of  readiness  and  preparedness  for  performing 
mission 


The  Operational  Need 


Centralized  resource  management  and  appropriate 
utilization  and  usage  of  it 

Multidimensional  management  (future  planning,  unit 
strategy,  short  term  objectives,  the  immediate  objectives) 

Initiating,  developing  and  implementation  management  of 
new  processes  and  technologies 

Balanced  planning  and  deploying  new  processes  and  tools 
improvements  and  new  technologies  in  a  measured  way 
that  will  quantify  the  improvement  vs.  expectations 

Information,  data  and  communication  security 


The  Operational  Need 


ch  person  working  in  the  implementation 
rganization  will  need  to  do  the  following: 

Access  the  processes  descriptions 
Understand  the  lifecycle  at  a  top  level 
Understand  in  detail  of  the  processes  that  he  or  she  performs 

In  addition,  managers  must  do  the  following: 

Understand  the  lifecycle  at  a  top  level 

Understand  the  leadership  change  management  expectations  in  detail 
Understand  how  to  lead  the  unit  using  the  new  processes 

Access  historical  measurement  data  for  all  processes  and  product 
versions  performance 

Support  implementation  of  new  processes  in  their  own  surroundings 
Remove  roadblocks  to  implementation 


Main  Risks  Areas  and  Impact 

(Example  Only) 


Risk  Class 

Risk 

Type 

Activity  or  Event 

Examples 

Mitigation 

Frequency 
&  Severity 

People 

Internal 

Unauthorized 

Activity 

Lack  of  skilled 
personnel 

Rogue  Trading 

High  employee  turnover 

Partially  insured 

People 

External 

Fraud 

Theft 

Partially  insured 

Systems 

Internal 

Model  Risk 

Mode  l/Methodology 
error 

Mark-to-model  error 

Technical  risk 
audit 

Improve  quality  of 
models/people 

Systems 

External 

Technology  Risk 

T  etecommunication 
failure 

Blackouts 

Contingency 

planning 

Insurance 

Processes 

Internal 

Transaction  Risk 

Execution  error 
Settlement  error 

Oocu  m  enta  li  o  n  /co  n  tract 
risk 

Improve 

processes 

Asset 

damage 

Internal 

Physical  asset 
risk 

Pipeline  Rupture 
Production  loss 
Unexpected  plant 
outage 

Partially  insured  1 

Contingency 

planning 

Asset 

damage 

External 

Physical  asset 
risk 

Uninsured  or 
irrecoverable  loss  or 
damage  to  assets 

Insurance 

The  Challenge 


This  situation  where  the  organization  is  running 

•  separate  process  improvements  on  different  parts  of  the 
system  /  product  lifecycle 

•  With  partial  overall  view  in  interactions  and 
handshakes  between  these  groups  is  introducing 
inefficient  usage  of 

resources, 

expensive  maintenance  of  duplicate  infrastructures 

and  Organizational  Sets  of  Standards  Processes  as  well  as 

assets, 

•  May  result  in  less  quality  and  impacting  the 
competitive  edge  with  their  global  counterparts. 


■  ■(CMJ)Model  Components  --  '’CM Ml  Staged* 


Common  Features 


'  (CMI)Model  Representation  Comparison 
'■(CMI)  Model  Tailoring 
L"j(CMI)Process  Areas 
■'(CMI)References 
'  (CMI)SpecJfic  Goals 
°  (CM  I  )S  pe  c  if  ic  P  ra dices 
0(CMI)Subgractices 

d{CMI)T ermmology  with  Special  Meaning 


°(CMI)Ability  to  Perform  (AB) 
:-(CMI)  Commitment  to  Perform  (CO) 


'  '(CMI)Directing  Implementation  (Dl) 
°(CM1)Verifying  Implementation  (VE) 


(C...^Q  (CMI)quantitatively  managed  process  Q  (CMI)optimizing  process  Q  (CMI)managed  process  i  (CM ^Characteristics of  Institutionalization  Q  (CMI)Common  Features 


Thought  Tags 


Reports  Calendar 


(CMI-CM)  Configuration  Management^ 


■(  ITI  L-S  S)Ser  vice  Support  ver  2.0 


H(ni L-SS-7)Configumtion  Management] 


•  (ITILS S7_t)Goat  of  Configuration  Management 
c,(ITIL-SS-7.10)lmpact  of  new  technology 
■'(ITIL-SS-7.1 1>Guidanceon  Configuration  Management 
*(ITIL-SS-7.2)Scope  of  Configuration  Management 
•  (ITI L-S S-7.3 )  Basic  concepts 
*(ITIL-SS-7.4)Benefits  and  possible  problems 
•  (ITI L-S S-7.5 )P  la n n i ng^ and  i m p Jementatio n 


am  L-S  S-7 _6)Acti  vities 
(ITIL-SS-7.7)Process  control 
(ITIL-SS-7.8}Relations  to  other  processes 
■  (ITI  L-S  S-7. 9  )Too  Is  specific  to  the  Configuration  Management  process 
■(ITIL-SS-7A)central  function  for  Change^  Configuration  and  Release  Management 
►(ITIL-SS-7B)Specific  responsibilities  of  the  Configuration  Management  team 


'  (CMI)Common  Features, n  (GMI)Model  Components  1 1  CMMI  Staged ISO  900  0:20 01  □  ITIL  LI  (lTIL-SS)Service  Support  ver  2.0  U  (ITIL-SS-7}Gonfiguration  Management 


%  1 1  (CMI-ML2)  Maturity  Level  2 -  M  (CMI-CM)  Configuration  ManagementpD  (CMI-CM-SGI)SG  1  Establish  Baselines^  (CMI-CM-SP  1.1JSP  1.1  Identify  Configuration  ItemSj 


Search 


(ITIL-SS-7.3.2)Configuration  identification  and  CIs* 

(ITI L-S S-7 .6.2 }Co nf  igu ratio n  identif i catio nf 
(ITIL-SS-7G)Suggested  Cl  attributes! 

(PRNC-1 9. 4.2)Confiq[u ration  identification! 

(PRNC-1 9.5.1 ) Configuration  management  cove../ 

(PRNC-1 9. 5.2)Choosing  the  level  of  product/- 
(SBQK-7.2)Software  Configuration  Identification! 

*  (C  M l-GM-S  P ik  1  )S P  l.l.lp, 
r  (CMI-CM-SP  j.  1  }SP  1.1.2s 
'*(  CMI-CM-SP 1 . 1  )SP  1.1.3. 


■(CMI-CM-SG1  )SG  1  Establish  Baselines. 


jCMI-CM-SPl.nSP  1.1  Identity  Configuration  Itemsj. 


•(GMI-CM-SP1 .2)SP  1.2  Establish  a  Configuration.. 
•(CMI-CM-SP  1 .3)SP  1 .3  Create  or  Release  Basel.. 


♦(CMI-CM-SP1.1  }SP  1.1.4. 

— •{CMi-CM-SPt  .1  )SP  1.1.5. 
•(CMI-CM-SP  U)SP  1.1. TWP. 


Notes 


Thought  Tags  Details  Search  Reports  Calendar 
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Arial 


□  Private 


identify  the  configuration  items,  components ,  and  related  work  products  that  will  be  placed  under 
configuration  management.  jTwsajGrof  sftot; 

Configuration  identification  is  the  selection,  creation,  and  specification  of  the  following:  faiss  gioispioi?4ioi] 
Products  that  are  delivered  to  the  customer 
Designated  internal  work  products 
Acquired  products 
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The  Approach  to  the  Solution 

Concept 

Best  practices  in  the  model  focus  on  activities  for 
providing  quality  services  to  the  customer  and  end 
users 


To  identify  improvement  targets  in  main  lifecycle 
areas  such  as  operations,  information,  governance, 
people  and  organizational  structure,  portfolios, 
project  execution,  and  finance 

Select  processes  that  are  critical  to  the  system 
success  such  as  stakeholder  management, 
technical  interfaces  and  integration 


The  Approach  to  the  Solution 

Concept 

Build  an  action  plan  composed  from  the  following  main 
steps 

•  Organizational  map 

•  Functional  team  and  groups  size  and  role  in  the  lifecycle 

•  Full  lifecycle  map 

•  Setting  improvement  targets 

•  Gap  analysis 

Suggesting  to  the  senior  management  to  address  the 
lifecycle  and  process  (as  a  whole)  as  a  complex  of  crossing 
interfaces  and  to  add  additional  content  to  the  lifecycle 
map  (as  a  layer) 


The  Conceptual  Solution 


Building  on  contingency  theory,  it  outlines  a 
comprehensive  framework  suggesting  a  fit 
between  the  level  of  Mission  interoperability  and 
environmental  as  well  as  internal  contingencies. 

Moving  from  the  current  environment  of  basic 
process  and  way  of  thinking  toward  a  more 
controlled  and  measured  process  to  reduce  the 
overwhelming  amount  of  information  that  build 
decisions 


The  Conceptual  Solution 


We  have  found  that  Maturity  Models  and  practices 
combined  with  some  other  industry  standards  and  methods 
as  a  new  integrated  approach  can  be  used  as  tools  to 
leverage  procedures  to  support  the  lifecycles  and  the 
organizational  business  objectives  and  capability,  readiness 
and  preparedness  to  achieve  improvement  and  excellence. 


H  The  Conceptual  Solution  -  1 

Building  on  contingency  theory,  it  outlines  a 
comprehensive  framework  suggesting  a  fit 
between  the  level  of  business  interoperability  and 
environmental  as  well  as  internal  contingencies. 

Moving  from  the  current  environment  of  basic 
processes  and  way  of  thinking  toward  a  more 
controlled  and  measured  set  of  processes  to  reduce 
the  overwhelming  amount  of  information  that  is 
now  required  to  build  decisions 
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•  Provide  the  ‘soft  and  intangible’  working  environments  and 
infrastructure  and  tools 

Process 

•  Provide  the  working  procedures  and  instructions,  which  assume 
to  guide  in  the  most  effective  way  how  to  use  the  facilities  and 
technology  to  achieve  the  business  objectives  by  the  people 

People 

•  Provide  the  individuals  that  build  the  teams  within  the 
organizational  units  and  groups,  that  perform  the  tasks  and 
activities  described  in  the  process 
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Risk  Evaluation  Checklist 


Business  Continuity  Plan  (BCP) 

Co  m  p  I  ete  A  ud  it  C  hec  k  I  i  st 


No 

Procedures 

Status  |  Notes  | 

i 

Determine  examination  scope  and  objectives  for 
reviewing  the  Business  Continuity  Plan  (BCP) 
program. 

2 

Determine  the  existence  of  an  appropriate 
enterprisewide  Business  Continuity  Plan  (BCP). 

3 

Determine  the  quality  of  Business  Continuity  Plan 
(BCP)  oversight  and  support  provided  by  the  board 
of  directors  and  senior  management. 

4 

Determine  whether  an  adequate  Business  Impact 
Analysis  (BIA)  and  risk  assessment  have  been 
completed. 

5 

Determine  whether  appropriate  risk  management 
over  the  Business  Continuity  Plan  (BCP)  process  is 
in  place. 

6 

Determine  whether  the  Business  Continuity  Plan 
(BCP)  include  appropriate  testing  to  ensure  the 
business  process  will  be  maintained,  resumed, 
and/or  recovered  as  intended. 

7 

Determine  whether  the  IT  environment  has  a  properly 
documented  Business  Continuity  plan  that 
complements  the  enterprise-wide  and  other 
departmental  Business  Continuity  plans. 

S 

Determine  whether  the  Business  Continuity  Plan 
(BCP)  include  appropriate  hardware  backup  and 
recovery. 

0 

Determine  whether  the  Business  Continuity  process 
includes  appropriate  data  and  application  software 
backup  and  recovery. 

10 

Determine  whether  the  Business  Continuity  Plan 
(BCP)  include  appropriate  preparation  to  ensure  the 
data  center  recovery  processes  will  work  as 
intended. 

11 

Determine  whether  the  Business  Continuity  Plan 
(BCP)  include  appropriate  security  procedures. 

12 

Determine  whether  the  Business  Continuity  Plan 
(BCP)  address  critical  outsourced  activities. 

13 

Discuss  corrective  action  and  communicate 

Data  RecoveryTemplates  and  Checklist 


Conducting  a  recovery  test 


Notes 


Activity 


N/A 


1 

Select  the  purpose  of  the  test.  What  aspects  of  the  plan  are  being 
evaluated? 

2 

Describe  the  objectives  of  the  test.  How  will  you  measure  successful 
a  ch  ieve  me  nt  o  f  th  e  o  h  ie  ctrves? 

3 

Meet  with  management  and  explain  the  test  and  objectives.  Gain  their 
aareement  and  suoport. 

4 

Have  management  announce  the  test  and  the  expected  completion  time. 

5 

Collect  test  results  at  the  end  of  the  test  period. 

6 

Evaluate  results.  Was  recovery  successful?  Why  or  why  not? 

7 

Determine  the  implications  of  the  test  results.  Does  successful  recovery 
in  a  simple  case  imply  successful  recovery  for  all  critical  jobs  in  the 
tolerable  outage  period? 

fl 

Make  recommendations  for  changes.  Call  for  responses  by  a  given 

9 

Notify  other  areas  of  results.  Include  users  and  auditors. 

10 

Change  the  disaster  recovery  plan  manual  as  necessary. 

Areas  to  be  tested 

Status  Notes 

N  Activity 

o  Y  N  N/A 

1 

Recovery  of  individual  application  systems  hy  using  files  and 
documentation  stored  off-site. 

!  2 

Reloading  of  system  tapes  and  performing  an  IPL  by  using  files  and 
documentation  stored  off-site. 

3 

Ability  to  process  on  a  different  computer. 

4 

Ability  of  management  to  determine  priority  of  systems  with  limited 
processing. 

5 

Ability  to  recover  and  process  successfully  without  key  people. 

6 

Ability  of  the  plan  to  clarify  areas  of  responsibility  and  the  chain  of 
command. 

BCP  TOC 


Business  Continuity  Planning  Components 


Getting  Sta  r te  d  Sectio  u  I 

L  As  sim  departmental  business  continuity  responsibilities. 

2.  Department  mission  and  business  functions  processes. 

3 .  Identification  md  evaluation  of  scenarios  .risks .  events  md 
threats . 


Dev  e  Loping  the  Pla  u  S  ectio  u  2 

4.  Document  recovery  plm3  to  recover  critical  functions  for  each 
scenario. 

0.  Determine  details  to  complete  tasks. 

6.  List  contact  information. 

7 .  List  neces  sarv  resources  md  reference  materials . 


Maintaining  the  Plan 

3.  Train  personnel  on  the  plan. 
9.  Test  (validate)  the  plm. 

HL  Maintain  the  plm. 
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2007  SCAMPI  A  Lessons  Learned 


y 


•  The  2007  SCAMPI  A  readiness  strategy  was  man-power  intensive 

•  The  3  day  Introduction  to  CMMI  course  did  not  map  to  LM  Aero 
terminology 

•  Including  dynamic  data  in  the  PHD  created  problems 

•  Lack  of  common  file  structure  across  the  program  PHD 
repositories  allowed  for  a  convoluted  mess 

•  Not  restricting  the  file  types  included  in  the  PHD  was  problematic 

•  Allowing  duplication  of  artifacts  within  a  program  PHD  created  CM 
issues 

•  Significant  planning  did  not  prevent  all  network  access  issues  for 
the  appraisal  team 

•  Comprehensive  documented  PHD  archival  process  is  important 


pud  -  Process  Implementation  Indicator  Description 
CM  -  Configuration  Management 
LM  Aero  -  Lockheed  Martin  Aeronautics 
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2010  SCAMPI  A  Improvements 


y 


•  A  more  centralized  approach  to  program  support  was  used 

•  More  comprehensive  use  of  “In-Briefings”  to  provide  affirmations 
for  Generic  Practices  reduced  the  artifacts  required 

•  LMASA  was  refined  to  target  more  specific  artifacts,  based  on  the 
standard  processes,  resulting  in  less  evidence  required 

•  “Gap  Events”  were  used  to  improve  LMASA  as  well  as  program 
understanding  of  evidence  needs 

•  The  PHD  Tool  was  changed  to  facilitate  appraisal  preparation: 

-  Incorporation  of  LMASA  Guidance  into  PHD  Tool 


LMASA  -  Lockheed  Martin  Aeronautics  Standard  Approach 
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Presentation  Topics 
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•  Opening  Brief  Standard 

•  PHD  Evidence  Standard 

•  Conclusions 


Copyright  ©  2010  Lockheed  Martin  Corporation 


Lockheed  Martin  Aeronautics  Company 


7 


Lockheed  Martin  Aeronautics  Standard  Approach - 7 

(LMASA)  Overview 


•  LMASA  provides  unambiguous  guidance  to  the  appraisal  projects 
in  how  LM  Aero  has  decided  to  present  our  evidence  to  the  SCAMPI 
team  during  our  appraisal 

•  LMASA  was  initially  developed  for  the  2007  SCAMPI  to  provide 
guidance  for  the  CMMI  Generic  Practices 

•  In  order  to  reduce  cost  for  the  2010  SCAMPI,  LMASA  was  expanded 
to  all  practices  of  the  CMMI.  It  also  was  refined  through  extensive 
review  to  tightly  align  with  our  standard  processes  and  their 
products. 

•  The  2007  SCAMPI  A  involved  3  appraisal  projects,  more  than  100 
participants,  across  3  widely-dispersed  sites  collecting  more  than 
3900  data  items. 

•  The  2010  SCAMPI  A  involved  five  projects  (two  focus,  three  non¬ 
focus)  across  the  same  three  sites,  but  required  only  about  2600 
data  items 
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Presentation  Topics 


•  Lockheed  Martin  Aeronautics  Overview 

•  LMASA  Overview 

•  Opening  Brief  Standard 

•  PHD  Evidence  Standard 

•  Conclusions 
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Opening  Brief  Standard 
-  Strategy  for  Affirmations  for  GPs 


7 


•  Provide  as  many  affirmations  for  GPs  in  the  Opening  Briefings  as 
possible 

•  Developed  a  template  for  the  projects  to  use  for  their  Opening 
Briefings 

-  Consistently  addressed  the  GPs  in  each  Program  Opening  Brief 

-  Coordinated  via  cross-program  dry  runs 


Copyright  ©  2010  Lockheed  Martin  Corporation 


Lockheed  Martin  Aeronautics  Company 


10 


Sample  of  Opening  Brief  Slide  Covering  GPs 
GP  2.2  Plan  the  Process 


Artifact 
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O 
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CL 

RSKM 

REQM 

D 

CO 
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CL 
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PPQA 

< 

< 
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< 
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Program  Management 
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X 

X 

X 

X 

X 

X 
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X 

Systems  Engineering 
Management  Plan 
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X 

X 

X 

X 

Configuration 
Management  Plan 

X 

Software  Development 
Plan  for  the  Mission 
Systems 

X 

X 

X 

X 

Requirements 
Management  Master  Plan 

X 

X 

Requirements  Work 
Package  Concept  of 
Operation 
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Presentation  Topics 
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•  Lockheed  Martin  Aeronautics  Overview 

•  LMASA  Overview 

•  Opening  Brief  Standard 

•  PHD  Evidence  Standard 

•  Conclusions 
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Developed  PIID  Evidence  Standard 


•  Reviewed  the  PIID  evidence  provided  from  all  projects  for  the  2007 
SCAMPI  to  develop  stream-lined  guidance  for  evidence  for  each 
practice  for  the  2010  SCAMPI 

-  Identified  specific  work  products  that  relate  to  LM  Aero  and  the 
specific  appraisal  projects 

-  Identified  where  evidence  was  needed  for  systems,  software 
and  hardware  examples 

•  Reviewed  the  LMASA  for  every  practice  with  the  projects  and  the 
process  owners  in  a  series  of  “Gap  Events” 

-  Validated  usability  of  LMASA 

-  Improved  the  project ‘s  understanding  of  what  the  ' best 
evidence ’  looked  like 

-  Determined  what  actions  if  necessary  were  needed  to  comply 
with  LM  Aero  best  practices  embodied  in  the  OSP 

•  Reviewed  LMASA  with  Lead  Appraiser  and  selected  Appraisal 
Team  Members  during  Class  C  Appraisal  August  2009 
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Structure  of  PIID  Evidence  Standard 


7 


•  CMMI  Identifiers 

-  Process  Area 

-  Specific  or  Generic  Practice  (SP/GP) 

-  Practice  Number 

•  Explanation  (Added) 

-  Text  to  explain  how  the  practice  is  implemented  in  the  Aero 
environment 

•  Guidance  (Refined) 

-  The  references  to  the  Aero  standard  processes  which  address 
this  practice  were  updated  and  made  more  explicit  (typically  to 
the  paragraph  level) 

•  Direct  and  Indirect  Evidence  (Refined) 

-  Updated  and  tied  more  directly  to  work  products  identified  in  the 
Aero  standard  processes 
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Samples  from  the  PIID  Evidence  Standard 


Process  Area 

Type  (GP/SP) 

Practice  Number 

Practice  Statemen 

Explanation 

Organizational 

Guidance 

Direct 

Evidence 

Indirect 

Evidence 

PA  ££ 

Type  T 

Practic  T 

Statemen  T 

Explanation 

Organizational  Guidance  T 

Direct  evidence 

Indirect  Evidencl [T 

PP 

SP 

1.2 

Establish  and 

For  Aero,  the  Capture  Team  will  analyze  the 

AC-5604,  Plan  and  Baseline  the 

BOEs  ("establish")  and  EACs 

Rationale  section 

maintain 

technical  solution  to  identify  the  program 

Program,  3.A.5.C  ("establish") 

("maintain")  showing  the 

of  BOE  Forms 

estimates  of 

attributes  of  work  products  and  tasks  that 

generation  of  estimates  based  on 

showing  identified 

the  attributes 

will  be  used  as  the  basis  to  estimate  effort. 

CPD-3032,  Estimating  and 

attributes  of  work  products  and 

attributes. 

of  the  work 

The  Capture  Team  will  then  use  the 

Pricing  Process 

tasks;  i.e.,  the  estimation  of 

products  and 

attributes  of  work  products,  tasks,  and 

attributes  such  as  "Source  Lines 

tasks. 

technical  requirements  (reference  CPD-3033, 

of  Code"  or  "engineering 

Contract  Technical  Requirements,  Proposal 

drawings"  which  are  then  used  to 

Technical  Requirements  and  CPD-3034, 

develop  estimates  of  cost  and 

Contract  Technical  Requirements-Proposal 

schedule.  These  could  be  called 

Planning)  to  develop  the  effort  estimate  in 

"Task  Sheets"  or  "Software  Task 

accordance  with  CPD-3032,  Estimating  and 

Sheets".  "Maintain"  can  also  be 

Pricing  Process  or  AC-5631,  Estimating  and 

shown  by  the  collection  of 

Pricing  Process.  The  resulting  estimates  and 

metrics  reflecting  the  attributes 

their  rationale  are  documented  in  Basis  of 

used  to  generate  the  estimate  in 

Estimate  (BOE)  sheets. 

the  BOE. 
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PIID  Evidence  Standard -  GP  Example 


y 


•  Example  -  GP  2.2  Plan 

-  Organizational  Directive 

•  One  record  that  points  to  where  in  your  process  a  plan  is 
required  (All  PAs) 

•  One  record  that  points  to  Section  1  .B  of  AC-5604  for  the  list 
of  all  plans,  one  of  which  is  yours  (even  if  it  is  not  visible  due 
to  being  part  of  another  plan,  such  as  the  SEMP)  (Project 
PAs  only) 

-  Program  Directive  =  Organizational  Directive 

•  If  you  obtained  a  tailoring  regarding  your  plan,  create  a 
record  that  shows  that  tailoring 

-  Direct  Evidence 

•  A  finalized,  approved  plan 
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PIID  Evidence  Standard-  GP  Example 


I  Process  Area:  Configuration  Ivknagement 
|  Category:  Support 

The  purpose  of  Configuration  Management  (CM)  is  to  establish  and  maintain  the  integrity  of  work  products  using  configuration  identification,  configuration  control,  configuration  status 
accounting,  and  configuration 

Standards  for  Direct  and 

|  Goal  GG  2  Institutionalise  a  Managed  Process 

The  process  is  institutionalised  as  a  managed  process.  “VlUtfllvtf 


the  process,  developing  1  re  work  products,  and  providing  the  services  of  the  configuration  management  process. 


Example  Indirect  Artifacts 


Typical  Work  Produc  ts 


[Pl’actke:  GP  2.4  Assign Responsibitity 
Assign  responsibility  and  authority forj 

I  Example  Direct  Artifacts 

I  LMASA:  Program  Ivknagement  Plan  estabhshing 
I  responsibility  for  CMby  roles;  Program  Org  Chart 
I  showing  people  assigned  to  CM  roles  (same  as  in  GP 
2.3) 

I  CMMI:  ■  Documentation  assigning  responsibility  for 
I  process  activities;  work  poducts,  or  services;  e.g.,  job 
I  descriptions,  or  plans  for  performing  the  process  (see 
|  GP2.2). 

■  Task  descriptions  and  activities  for  defined  roles. 

I  Appraiser  Considerations 

LMASA  Organizational  Guidance:  AC-5604,  Plan  and  Baseline  the  Program,  3D  □  AC-5605,  Organize  the  Program,  3.C,  3.D.5.eD  □  AC-5607,  Monitor  and  Control  Program  Performance 

LMASA  Considerations:  For  Aero,  AC-5604  establishes  respnsibility  for  various  elements  of  the  program  plan.  AC-5605  specificallyaddiesses  the  assignment  of  personnel  for  prog  rams . 
Section  3.C  describes  the  assignment  of  management  personnel  to  the  lowest  tier.  Section  3. D.5.e  describes  the  assignment  of  personnel  to  roles  within  each  IPT  to  the  various  team  members. 
Responsibilities,  roles,  and  personnel  assignments  are  covered  in  the  Frog  ram  Manage  merit  Plan  and  the  Frog  ram  Organization  Charts;  as  well  as  in  IPT  Charters. 

CMMI  Considerations:  ■  These  activities  maybe  distributed  across  different  groups  within  the  organization  (e.g.  systems,  software,  CM  group). 

■  Responsibility  may  change  as  development  progresses  across  the  life  cycle. 


LMASA:  Affirmations  from  Program  In-Brief. 

CMMI:  ■  Assignment  is  often  in  the  project  plan  or 
configuration  management  plan 


Other  contextual  help 
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PHD  Evidence  Standard-  SP  Example 
PP  SP  1.2 


•  Example  Direct  Artifacts 

-  LMASA:  BOEs  (‘establish”)  and  EACs  (‘‘maintain”)  showing  the 
generation  of  estimates  based  on  attributes  of  work  products 
and  tasks;  i.e.  the  estimation  of  attributes  such  as  “Source 
Lines  of  Code”  or  “engineering  drawings”  which  are  then  used 
to  develop  estimates  of  cost  and  schedule.  These  could  be 
called  “Task  Sheets”  or  “Software  Task  Sheets.”  “Maintain” 
can  also  be  shown  by  the  collection  of  metrics  reflecting  the 
attributes  used  to  generate  the  estimate  in  BOE. 

•  Example  Indirect  Artifacts 

-  LMASA:  Rationale  section  of  the  BOE  Forms  showing  identified 
attributes. 
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PIID  Evidence  Standard-  SP  Example 


Process  Ana:  P^jKtPlannin? 

Category:  ftnjBd  Hb^ui 

Tit  pspoa  of  Pry  tot  P]rT-iaj  fPP)  l;  :□  K:i,tlib  md  mzinbrin  flans  i in  d:nBt  pre^arl  iccn.ib.gE. 


Goal:  --'S  1  ErtAEifc  EsimiabK 

Esbzanos  p r :-j c-: h  ['i! —  f  p aizmocor:  nt  asubLidod  and.  mainuingd. 


Standards  for  Direct  and 
Indirect  evidence 


Practice-:  5P  1*  ErtdHAEsdnnataiofTOiikftiidiietj 


EstabLd  and  zaiiz.:  linos  a  malt:  of  do  a 


:  dug  wod^  pcndnriE  iz.d  IieIe. 


Eiample  Indirect  Artifact! 

1A1A&A  RiannzJo  5  tea  on  of  BOE  farms  sticking 
idtraStd  jrailnlt: 


CMMk  [1.  TodmiciJ  appcnzob] 

P  Six:  and  roaqplKiatY  af::  d;,  and  .red  juodidbs] 
P  Er'daiaiiz.c  mxlols] 

"  E  idnii line  ::■:■] e.  zJEOxidmE,  and  pcaiaduras 

■  Cf«uab.oiaL  dgEmbons  {a. a.,  fcjigdur:  orihar_-aj  tor 
g^sbLrldos  ud  dnrunanbna  dig  ssamaltiofdo 
arailn1t:ofdo  work  pcoduols  sod  cast 

■  Bisas  ofEEliiEaEgE  (EQEs) 

1  Usa  of  validated  modal:. 

■  Us ei  afinodalk  -!■:  ut  oiUbr:::-!  wid  kistecLoal 


T>|icanVork  Product! 

T:cbniczl  ippcnzzb 

S-iza  uudkomflaitiiy  of  lasts  and  ivi-ik  pm  duds 
Estunianfl  medals 
Altribiug  ErElimatBE 


Eiample  Direct  Artifacts 

LHASA:  BOEe  f trzblsb^  md  EACe  fmzjnferin^ 
stc'vinj  da  sGoar;n.nn  of  aftmuriM  basad  an  iTlributg: 
of  woct  products  zzjA  tzaks;  La,,  tig  Kama  box.  of 
zlteLbuigs  snebzs  'SovnLnm  nfCndiT'oT 
giimcggrioj  dczTiin.:;: '  tv!h:L  it:  clan  ustd  be  davolof 
animates  ofeesc  and  ;eboivJt  ThastemiJdtg  raLLad 
rTadE  Shads’  nr  ’Eoftwun  Task  &1m1  e  1  '7r[iin1iinr 
mi  also  ba  Ebo'j.11  try  ibt  eolLgranu  of  marries 
xofltetinE  ibt  auibult:  j:td  cangratg  Ihg  oscxnzlt  m 
daB M. 

CMHt  [4.  Ateribvte  KtmuJK] 

■  Ee  timings  eflha  imp-ala:  of  fbo  wnrk  product:  and 
tasks  (a^_  eim) 

■  Esbmugs,  as  appropriate,  of  hboc.  mzehiDMy. 

matenals.  and.  matbedr  ihz:  niD  1b  itqnicod  by  cbo 
pcojKE  — 

■  Ee  timings  coTidnu  hisie-jT. 

Appraiser  Can  yds  rations 

LMASA  3njzmzahoi.il.  GuidsnoK  AC-  5dC4.  Flan  :_d  Eiwlmo  da  Piogjun.  3  A  lie  i‘  triz'jJsb.' ;  CPD-30J  1 .  Eslimabnc  ind  Priemc  Ptocacs 

US  LAS  A  Z  □QSL'dgrinzns:  Fox  Aocn,  tbg  Cupbict  Taim  yt_1  izaLyzu  da  teehzi:aL  soLiaon  to  :d:oc.f>-'  do  f  CDmam  almb_:os  of  Talk  pcodiids  and.  basks  du  mil  It  -tad  as  tbg  lids  to 
tolimate  of  ton:  Tla  Capbut  Tana  veil  tbgn  11st  ibt  atenbnte:  of  work  products^  :isks.  ind  :odmLc  j  itquiranugat;  >;e;:ectc:b  Z  7L- 3035.  Conlriot  TtcbmoiJ  EAqniioaugnls.  Ptoposii 
T  tohnlozl  EsKpmomgats  and  CPD-3054,  Coouzoc  TardiraL  ^oqiiunnmL:E-]lopoaii  Pluming;)  In  dnivlop  do  a  tor  todmalB  in  zoomdanoo  wid  CPE-3032,  ZErimzang  and  Pclcing  Pjocbso 


Other  contextual  help 


L  hurtdiv,  Nortmlac  05,  2G09 


Pact  1  of  2 
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PHD  Repository  Standard  - 
Screen  Shot  from  PIID  Tool 


Practice  Implementation  Indicator  Descriptions 


Practice:  PP  SP  1.2 

Classification:  Implementation  Direct — 

More  Info 

Project  Planning 

CM  Ml  practice 

Establish  Estimates  of  Work  Product  and  Task  Atti\  idSPltifiST 

and  tasks.  S 

y/ntain  estimates  of  the  attributes  of  the  work  products 

Document  No.:  N/A 

^Adequacy:  Fully  Satisfied 

v  ’“Owner  Org:  F-16  MPEC 

^Document  Title: 
^Section: 

Location: 

*Link: 

^Rationale: 


BOEs  -  Source  Lines  of  Code  Estimate 


Sec  2. 1.2.1  Pages  39-40 


file  server 
hyperlink  not  a 
web  page 


'i  'i  FT  WSS03\  F- 16  USAF  EPAF  TEAM'1.  Mission  Planning^  16  UPC  MPE  DO  S'-.CMMI  Evidence'1.  PP'1.  SPs\  Direi: 


Edit 


Chk 


The  Basis  of  Estimate  (BOE)  in  Section  2. 1.2.1  Pg  39-40  demonstrates  the  source  lines  of  code  that  is  a  key 
attribute  (Size)  of  the  MPEC  work  product. 


Comments: 


PIID  Record  State:  Reviewed 


ID: 


23346  Go  To 


*  -  Required  Fields. 


Show  pending 

records  only 

M 

► 

►1 

|  Submit 

|  Accent 

|  Refect 

|  Refresh  | 

Retire 

1  Edit  CA 

Show  retired 
records 


Record  1  of  1. 


IP 


& 
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Presentation  Topics 
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•  Lockheed  Martin  Aeronautics  Overview 

•  LMASA  Overview 

•  PHD  Evidence  Standard 

•  PHD  Repository  Standard 

•  Opening  Brief  Standard 

•  Conclusions 
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Reduced  Cost  of  Appraisal 


7 


120% 
1 00% 
80% 
60% 
40% 
20% 
0% 


2010  Appraisal  accomplished 

at  20%  the  cost  of  the  2007 
Appraisal 

1 _ 1 

1 

- 1 - 

i  i 

2007  Actuals  2008  Estimate  2009  Estimate  2010  Actuals 

%  of  2007 
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Conclusions 


7 


•  Use  of  improved  evidence  collection  tools  and  methods  greatly 
reduced  the  cost  of  appraisal  preparation 

-  Initial  estimate  for  preparation  for  2010  appraisal  was  reduced 
to  52%  of  the  cost  of  the  2007  appraisal 

-  A  later  estimate  for  preparation  for  2010  appraisal  was  reduced 
to  %  of  the  cost  of  the  2007  appraisal, 

-  Actual  cost  of  preparation  for  the  2010  appraisal  was  of  the 

actual  cost  for  the  2007  appraisal,  despite  the  inclusion  of  two 
additional  projects 

•  Development  of  the  LMASA  also  resulted  in  corrections  and 
improvements  to  the  standard  processes 

•  Use  of  LMASA  by  the  programs  resulted  in  improved  adherence  to 
the  standard  process 
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Questions? 
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Systems  Assurance,  The  Global  Supply  Chain,  and 
Efforts  To  Increase  Communication  Between 
Acquisition  and  Development 


Michele  Moss 
NDIACMMI  Technology  Conference 

November  17,  2010 


|  Table  Of  Contents 


►  Globalization  Challenges 


►  Understanding  The  Problem 


►  Working  Towards  A  Solution 


Globalization  brings  challenges 


* 


Technology  Is  A  Focal  Point  Of  Attacks 


Who  is 
behind  data 
breaches? 

74%  resulted  from  external  sources  (+1%). 

20%  were  caused  bv  insiders  (+2%). 

I  32%  implicated  business  partners  (-7%). 

39%  involved  multiple  parties  (+9%). 

How  do 

breaches 

occur? 

7%  were  aided  by  significant  errors  (<>). 

64%  resulted  from  hacking  (+5%). 

|  38%  utilized  malware  (+7%. 

22%  involved  privilege  misuse  (+7%). 

9%  occurred  via  physical  attacks  (+7%). 

*  Source  -  2009  Verizon  Data  Breach  Investigations  Report 


According  to  an  article  in  the  May  2010 
National  Defense  Magazine,  well  funded 
nation  states  and  terrorist  organizations 
are  engaging  in  cyber  attacks  against  US 
government  systems.  Examples  of  those 
include  44,000  Turkish  teenagers  in  a 
military  style  community  of  hackers 
learning  from  each  other. 

There  are  also  100,000  hackers  learning 
from  each  other  in  Saudi  Arabia,  40,000 
in  Iraq,  and  over  400,000  in  China. 


3 


3 


Increased  Priority  for  Program  Protection 

►  Threats :  Nation-state,  terrorist,  criminal,  rogue  developer  who: 

-  Gain  control  of  systems  through  supply  chain  opportunities 

-  Exploit  vulnerabilities  remotely 

►  Vulnerabilities :  All  systems,  networks,  applications 

-  Intentionally  implanted  logic  (e.g.,  back  doors,  logic  bombs,  spyware) 

-  Unintentional  vulnerabilities  maliciously  exploited  (e.g.,  poor  quality  or  fragile 
code) 

►  Consequences :  Stolen  critical  data  &  technology;  corruption,  denial  of  critical  warfighting  functionality 

Today’s  acquisition  environment  drives  the  increased  emphasis: 

Then  Now 

Standalone  systems  »>  Networked  systems 

Some  software  functions  »>  Software-intensive 

Known  supply  base  »>  Prime  Integrator,  hundreds  of  suppliers 


Source:  Source:  September  28,2010  SwA  Forum,  DoD  Trusted  Defense  Systems,  Ms.  Kristen  Baldwin,  DDR&E/Systems  Engineering 


“U.S.  charges  Florida  pair  with  selling  counterfeit  computer 
chips  from  China  to  the  U.S.  Navy  and  military” 


INCIDENT: 

On  September  14,  2010  Federal  prosecutors  in 
Washington  unsealed  charges  accusing  a  Florida  pair 
of  selling  more  than  59,000  counterfeit  computer 
microchips  from  China  to  the  U.S.  Navy  and  other 
clients  for  military  use  aboard  American  warships, 
fighter  planes,  missile  and  antimissile  systems.  Wren, 
owner  of  VisionTech  Components  and  related 
companies,  and  McCloskey,  an  administrator,  were 
charged  with  conspiracy,  trafficking  in  counterfeit 
goods  and  mail  fraud. 

MITIGATION: 

In  January  the  Commerce  Department  reported  that 
the  number  of  counterfeit  incidents  discovered  by  the 
military  and  its  suppliers  more  than  doubled  between 
2005  to  2008,  to  more  than  9,356  cases.  Meanwhile, 
lawmakers  and  congressional  investigators  have  called 
on  the  Pentagon  and  law-enforcement  agencies  to 
combat  the  problem  more  aggressively. 


Courtesy  of  Don  Davidson,  OSD  TMSN  , Chief  of  Outreach  and  Standardization 


IMPACT: 

The  case  marked  the  latest  effort  by  U.S.  authorities  to 
stem  the  flow  of  fake  electronics  into  the  U.S.  military 
supply  chain,  as  warnings  mount  that  fake  chips  could 
be  defective  or  "electronic  Trojan  horses"  that  would 
allow  hackers  to  disable  them  or  track  their  use.  Several 
recent  government  reports  warn  that  computer  chips 
marked  with  false  brands  or  mislabeled  as  military-grade 
may  include  imperfections  that  could  cripple  or  degrade 
weapons  systems  in  combat  or  over  time. 


http://www.washingtonpost.com/wp- 
dyn/content/article/201 0/09/1 4/AR201 0091 406468.html 


“Recalled  BlackBerry  Batteries-  Yet  another  wake-up  call  on 
counterfeit  parts  and  product  recall  mitigation” 


INCIDENT: 

On  August  10,  2010  the  U.S  Consumer  Product 
Safety  Commission  announced  that  about  470,000 
BlackBerry  batteries  distributed  by  Asurion  were  being 
recalled  due  to  an  overheating  and  safety 
problem.  According  to  the  recall  notice,  the  batteries 
in  question  were  counterfeit,  and  “these  batteries  were 
used  across  virtually  all  modes  of  refurbished 
BlackBerry  devices  distributed  by  Asurion  prior  to 
November  1,  2009.  ” 

MITIGATION: 

Consumers  who  received  refurbished  BlackBerry 
devices  through  Asurion  prior  to  November  1 , 2009 
were  advised  to  immediately  stop  using  the  product 
and  contact  Asurion  for  a  replacement  product. 

Asurion  is  directly  contacting  known  consumers  with 
the  affected  batteries  to  notify  them  of  this  recall. 


Courtesy  of  Don  Davidson,  OSD  TMSN  , Chief  of  Outreach  and 
Standardization 


IMPACT: 

The  counterfeit  batteries  can  overheat,  posing  burn  and 
fire  hazards.  Asurion  has  received  two  reports  of 
counterfeit  BlackBerry®-branded  batteries  overheating, 
causing  minor  burns  to  a  consumer's  finger  and  minor 
property  damage  to  a  sofa  and  car  seat. 


http://www.theferrarigroup.com/blog1/2010/08/11/recalled- 

blackberry-batteries-yet-another-wake-up-call-on-counterfeit- 

parts/ 
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►  Globalization  Challenges 


►  Understanding  The  Problem 
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ICT  SCRM  And  SwA  Are  Complex  Multi-Disciplinary 
Challenges 


Software  Assurance  (SwA) 


Project 

Management 


Information 

Assurance 


System 

Engineering 


Test  & 
Evaluation 


Safety  and 
Security 


Information 

Systems 

Security 

Engineering 


Source:  https://buildsecuritvin.us-cert.gov/swa/procresrc.html 


Communication  Across  Organizational  Stakeholders  Is 
Critical  to  addressing  ICT  SCRM  and  SwA  Challenges 


Define  Business  Goal 


Development  Organization 

DO  1  Establish  the  assurance 
resources  to  achieve  key 
business  objectives 

DO  2  Establish  the  environment  to 
sustain  the  assurance 
program  within  the 
organization 


I* 


Development  Project 

DP  1  Identify  and  manage  risks 
due  to  vulnerabilities 
throughout  the  product  and 
system  lifecycle 

DP  2  Establish  and  maintain 

assurance  support  from  the 
project 

DP  3  Protect  project  and 

organizational  assets 


Prioritize 
funds  and 
manage  risks 


Enterprise  Assurance 
Support 

ES  1  Establish  and  maintain 
organizational  culture 
where  assurance  is  an 
integral  part  of  achieving 
the  mission 

ES  2  Establish  and  maintain  the 
ability  to  support 
continued  delivery  of 
assurance  capabilities 

ES  3  Monitor  and  improve 

enterprise  support  to  IT 
assets 


Acquisition  and  Supplier 
Management 

AM  1  Select;  manage,  and  use 
effective  suppliers  and 
third  party  applications 
based  upon  their 
assurance  capabilities. 


Development  Engineering 

DE  1  Establish  assurance 
requirements 

DE  2  Create  IT  solutions  with 
integrated  business 
objectives  and  assurance 

DE  3  Verify  and  Validate  an 
implementation  for 
assurance 


Enable 
Resilient 
Technology 


Sustained 
environment  to 
achieve 
business  goals 
through 
technology 


The  Assurance  PRM  Is  A  Holistic  Framework  that  connects  CMMI  and  RMM  to  facilitate  communication 

https://buildsecuritvin.us-cert.gov/swa/proself  assm.html 


A  Resilient  Technology  Best  Practices  Cross  Walk 


You  have  been  asked  to  ensure  that  the 

OWASP  Top  Ten  (an  assurance  coding  ^ou  can  '°°*<  a*  OSAMM  for 

Standard)  are  not  in  the  Code  guidance  on  how  to  do  it 

_ _ J  ' - 


nity’s  Assurance  Process  Reference  Model  -  Initial  Map^ 


[n  the  following  tab.  jnce::  are  intended  to  include  system  and  software  assurance,  information  assurance,  and  ■  t  of  the 

_ . _  bu  si  ness/mission  functions  supported  by  systems  and  software.  _ 


Goal 

;e 

AFCMMI 

BSIMM 

CMMI-ACQ 

CMMI-DEV 

t 

MM 

RMM 

DE  2  Create  IT  solutions 
with  integrated  business 
objectives  and  (assurance 

Develop  alte  and  selection  criteria  for 

mission  and  in.  ance. 

AFTS  SP  1.1.1 
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ATM  SG2 

TS  SGI 

1A 

RTSE:SG 

1  -  SG2 

3FD1.2 

AVAL  SG2 

MB 

KIM:SG2, 
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Architecl  foe  mission  a  rmation  assurance. 

AFTSSP  2.1.1 

SFD2.1 

ATM  SG2 

TS  SG2 

■:  A2A 

RTSE:SG 

3 

SFD2.3 

AVAL  SG2 

TS  SG2 

;A2E 

Design  for  missior^and  info,  mation  assurance. 

AFTS  SP  2.1.2 

SFD2.1 

TS  SG2 

IjjipWflSflPfhe  mission  and  infonflTlBtt^aij^nce 

desion  s  of  the  product  com oonents . 

AFTS  SP 3.1.1 

TS  SG3 

Identify  deviations  from  mission  and  information  ^ 
assurance  coding  standards.  Implement  appropriate  1 
mitigation  to  meet  defined  mission  and  information  J 
assurance  objectives. 

AFTSSP  3.1 .2  I 

r  CR1.4^ 

AVER  SG3 

TS  SG3 

C  R2  A  ^ 

^TSE:SG 

\  2 

CR2.3  J 

CR2B 

?TSE:SG 

/  3 

X^CR3.l/ 

^CR3A^> 

https://buildsecuritvin.us-cert.gov/swa/proself  assm.html 


Mapping  to  the  Engineering  for  System  Assurance,  vl.O  is  in 
progress 


►  NDIA/DoD  guidebook  providing  process 
and  technology  guidance  to  increase 
the  level  of  system  assurance. 

►  Intended  primarily  to  aid  program 
managers  (PMs)  and  systems 
engineers  (SEs)  who  are  seeking 
guidance  on  how  to  incorporate 
assurance  measures  into  their  system 
life  cycles. 


Engineering  for 
System  Assurance 

Version  1.0 

National  Defense  Industrial  Association 
System  Assurance  Committee 

MMIL 

iliFiii 

«  i)i  ■‘iiiiiir.il  MnMfcl  A  in  ifeMil  iii1 


http://www.acq.osd.mil/sse/ssa/docs/SA-Guidebook-v1-Oct2QQ8.pdf 


Enterprise  Leadership  and  Resilient  Technology 


Business 

Functions 


Define  Business  Goals 

Development  Organization 


Sustained  environment  to  achieve 
business  goals  through  technology 

Enterprise  Assurance  Support 


Enable  Resilient  Technology 
Development  Project 
Development  Engineering 


Prioritize  funds  and  manage 
risks 


Business,  Processes 


*  ^  ^  4  -is 

i  S"V~  ~  -  C  ,  -  > 

Assets  in  Production 


peop/e 


info 


tech 


faciliti 


Service 
Mission 


Adapted  from:  Source:  November  2009  SwA  Forum-Evolution 
in  SwA  Processes  Panel  -  David  White,  SEI 


SwA,  SCRM,  And  Continuous  Improvement  Contribute  To 
Operational  Resilience 


Are  we  being 
attacked? 


Compliance  ^ 


OVAL 

SCAP 


Incident 

^  Management 
and  Control 


Who  is  attacking  and 
what  do  they  want? 


Are  we  at 
risk? 


Controls 


Applied  to 


Asset 

Management 


Adapted  from  September  2010  SwA  Forum,  CERT  RMM  for  Assurance  ,  Lisa  Young,  SEI 


ICT  Supply  Chain  Assurance 

An  I  AT  AC  State-of-the-Art  Report 


The  following  link  is  available  to  personnel  accessing  from 
within  a  .mil  or  .gov  domain: 

URL:  http://iac.dtic.mil/iatac/pdf/supplv  chain.pdf 

You  may  also  contact  IATAC  directly  to  obtain  access  to  this 
report.  The  easiest  way  for  you  and  the  IATAC  team  to  get 
you  the  report  is  for  you  to 

Information  Assurance  Technology  Analysis  Center 
(IATAC) 

Email:  iatac@dtic.mil 

URL:  http://iac.dtic.mil/iatac/ 


SwuridyfltSlf 

Management  V',  ' 

|  pi  pr  nuptial* 
rht»-iflrc.pf  |idi  ■ 

Supply  Chain 
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SAFECode  (www.safecode.org) 


►  SAFECode  is  a  global,  industry-led  effort  to  identify  and 
promote  best  practices  for  developing  and  delivering 
more  secure  and  reliable  software,  hardware  and 
services 

►  White  papers 

-  Software  Assurance:  An  Overview  of  Current 
Industry  Best  Practices 

-  Fundamental  Practices  for  Secure  Software 
Development 

-  Security  Engineering  Training:  A  Framework  for 
Corporate  Training  Programs  on  the  Principles  of 
Secure  Software  Development 

-  Framework  for  Software  Supply  Chain  Integrity 

-  Software  Integrity  Controls:  An  Assurance-Based 
Approach  to  Minimizing  Risks  in  the  Software  Supply 
Chain 


Describing  the  Software  Supply  Chain 


►  Sophisticated  IT  solutions  are  composed  of  a  collection  of  components 


► 


► 


Each  component  or  its  parts  can  be: 

-  Developed  by  its  supplier  or  on  that  supplier’s 
behalf  by  their  subcontractors;  or 

-  Licensed  to  the  supplier  by  another  vendor  or 
obtained  from  Open  Source  repositories;  or 

-  Acquired  outright  by  the  supplier 


Supplier 

Sourcing 

•  Procurement 


Product  Development 
and  Testing 

•  Environment 

•  Personnel 

•  Software  Development 


Product 

Delivery 

•  Distribution 

•  Maintenance 


Regardless  of  the  development  scenario,  each  software 
supplier  in  the  supply  chain  must  manage  three  sets  of  controls: 


-  1.  Supplier  Sourcing  —  Select  the  suppliers,  establish  the  specification  for  the 
supplier’s  deliverables,  and  receive  software/hardware  deliverables  from  the  suppliers; 


-  2.  Product  Development  and  Testing  —  Build,  assemble,  integrate  and  test 
components  and  finalize  for  delivery;  and, 

-  3.  Product  Delivery  —  Deliver  and  maintain  their  product  components  to  their 
customer. 


Source  -  SAFECode:  Framework  for  Software  Supply  Chain  Integrity 


Software  Supply  Chain  Staircase 

►  Figuratively,  an  IT  solution  supply  chain  can 
resemble  a  collection  of  staircases  involving 
the  successive  transmission  of  software 
components  from  a  supplier  to  its  customer 


►  In  this  figure,  components  move  along  the 
“staircase”  supply  chain  as  they  are  handed 


off  from  one  supplier  to  the  next.  At  each  step 
a  supplier  controls  three  links  in  the  supply 
chain: 


1 .  Goods  received  from  suppliers; 

2.  Their  product  production;  and 

3.  What  is  delivered  to  their 


Her  2  Supplier 


Software  Supply  Chain  Staircase 
^  Acceptance  Test 


Release  Test 


customers 


Project  Lifecycles 


Source  -  SAFECode:  Framework  for  Software  Supply  Chain  Integrity 


Fundamental  Software  Supply  Chain  Integrity  Controls 


►  Software  supply  chain  integrity  controls  address  the  access,  storage  and  handling  of 
development  assets  throughout  the  supply  chain  -  supplier  sourcing,  product 
development  and  testing,  and  product  delivery. 

►  Some  fundamental  software  supply  chain  integrity  controls,  derived  from  established 
security  and  integrity  principles,  include: 


Control  Title 

Description 

Chain  of  Custody 

The  confidence  that  each  change  and  handoff  made  during  the  source  code’s  lifetime  is 
authorized,  transparent  and  verifiable 

Least  Privilege  Access 

Personnel  can  access  critical  data  with  only  the  privileges  needed  to  do  their  jobs. 

Separation  of  Duties 

Personnel  cannot  unilaterally  change  data,  nor  unilaterally  control  the  development  process 

Tamper  Resistance  and  Evidence 

Attempts  to  tamper  are  obstructed,  and  when  they  occur  they  are  evident  and  reversible. 

Persistent  Protection 

Critical  data  is  protected  in  ways  that  remain  effective  even  if  removed  from  the 
development  location. 

Compliance  Management 

The  success  of  the  protections  can  be  continually  and  independently  confirmed 

Code  Testing  and  Verification 

Methods  for  code  inspection  are  applied  and  suspicious  code  is  detected. 

Source  -  SAFECode:  Framework  for  Software  Supply  Chain  Integrity 
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Software  Sourcing  Controls:  Contractual 


► 


► 


A  vendor’s  engagement  with  a  supplier  is  governed  by  a 
written  agreement,  for  example  a  license  or  a  contract 

The  written  agreement  must  explicitly  state  the  vendors  an_. 
supplier’s  expectations,  as  well  as  the  consequences  of  any 
non-compliance  with  the  terms  of  the  agreement 

Software  sourcing  controls: 


-  Defined  Expectations  —  Clear  language  regarding  the  requirements  to  be  met  by  the  code 
and  the  development  environment  should  be  set  forth 


-  Ownership  and  Responsibilities  —  1C  and  responsibilities  for  protecting  the  code  and 
development  environment  must  be  articulated 


-  Vulnerability  Response  —  How  well  is  the  supplier  equipped  to  collect  input  on 
vulnerabilities  from  reputable  sources  and  appropriate  remedies 

-  Security  Training  —  How  well  is  the  partner  able  to  effectively  train  its  developers  on 
security  development  practices 

-  Open  Source  Software  —  The  use  of  OSS  presents  alternative  challenges  in  the  context  of 
supply  chain  integrity 

Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 


Software  Sourcing  Controls:  Technical 


►  Secure  Transfer  —  Delivered  code  should  be 
transferred  securely,  using  authenticated  endpoints 
and  encrypted  sessions 


►  Sharing  of  System  and  Network  Resources  —  The 
digital  identities  a  vendor  issues  to  suppliers  to  enable 

access  to  the  vendor's  network  and  resources  should  be  established  with  strong  controls 
enforced  to  limit  access  to  only  those  resources  needed  to  perform  the  supplier's  role 

►  Malware  Scanning  —  Supplier  content  to  be  transmitted  to  the  vendor  should  be  scanned 
for  malware  using  the  most  recent  malware  signature  files  and  more  than  on  commercial 
scanning  engine 

►  Secure  Storage  —  Source  code  should  be  stored  securely  with  need-to-know  access 
controls  applied 

►  Code  Exchange  —  Processes  using  digitally  signed  packages  and  verifiable  checksums  or 
hashes  should  be  in  place  to  ensure  that  received  code  is  complete  and  authentic 


Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 
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Software  Development  and  Testing  Controls:  Technical 


►  People  Security  —  Ensure  that  adequate  background 
checks  are  performed,  that  roles  and  responsibilities 
and  access  rights  are  clearly  defined,  and  that 
segregation  of  duties  and  controlled  automated 
processes  are  applied. 


►  Physical  security  —  Building  security  and  physical  access  control  should  be  applied  to 
development  locations  and  code  repositories  and  periodically  re-assessed  using  a  risk- 
based  process 


►  Network  Security  —  Network  security  standards  should  be  established  and  applied  using  a 
risk-based  process  for  code-related  assets 


►  Code  Repository  Security  —  All  code-related  assets  should  be  housed  in  source  code 
repositories  to  enable  additional  attention  to  security  and  access  control 


►  Build  Environment  Security  —  Build  environments  should  be  as  automated  as  much 
possible  to  minimize  the  opportunity  for  human  intervention  in  the  regular  build  process 


Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 


Software  Development  and  Testing  Controls:  Security  Testing 


►  Peer  Review  \  Manual  Inspection  — 

-  Are  not  often  popular  given  issues  of  scalability, 
but  automated  tools  can  enable  some  scalability 
by  collecting  and  processing  more  artifacts  in 
preparation  for  peers  performing  a  focused  review 


-  Also,  when  teams  are  assigned  to  work  together  on  code  files,  an  important 

dynamic  is  present  whereby  reviewers  can  more  readily  identify  code  that  does  not 
belong  within  a  code  set 


►  Testing  for  Secure  Code  — 

-  The  size  of  the  code  base  for  many  software  projects  today  requires  automated  code 
review  and  testing  tools 

-  Building  these  tests  to  run  in  a  repeatable  automated  manner  increases  the  assurance 
that  they  will  be  performed  and  analyzed  often 

-  These  tools  include:  static  code  analysis  tools,  vulnerability  scanners,  binary  code 
analysis  tools,  malware  detection  tools,  security  compliance  validation  tools,  and  code 
coverage  tools 

Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 
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Software  Delivery  Controls:  Publishing  and  Dissemination 


►  Covers  new  product  delivery  and  the  delivery  of 
maintenance  patches 

►  Not  always  last  stage  of  the  supply  chain 

►  As  software  components  leave  the  supplier, 
software  integrity  and  authenticity  become  a  shared 
responsibility  between  the  supplier  and  customer. 


Software 

Sourcing 

•  Procurement 

Software  Development 
and  Testing 

•  Environment 

•  Personnel 

•  Software  Development 

Software 

Delivery 

•  Distribution 

•  Sustainment 


►  Malware  Scanning  —  Products  should  be  scanned  for  malware  using  the  most  recent 
malware  signature  files  and  more  than  one  commercial  scanning  engine 

►  Code  Signing  —  The  software  vendor's  product  should  be  strongly  digitally  marked  with  the 
software  vendor's  identity  in  a  way  that  can't  be  altered,  yet  may  be  verified  by  the 
customers 


►  Delivery  —  A  vendor's  process  for  delivering  products  both  online  and  through  distributions 
using  physical  and  electronic  media  should  be  secured;  information  on  code  signing  and 
checksums  should  be  available  to  customers. 


►  Transfer  —  Vendors  should  transfer  products  in  such  a  way  that  the  receiver  can  confirm 
that  the  product  is  coming  from  the  software  vendor 

Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 
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Software  Delivery  Controls:  Authenticity 


►  Over  one  in  five  software  packages  is  counterfeit 
or  pirated1 

►  Authenticity  is  a  core  element  of  software  assurance 

►  The  risk  of  counterfeit  software  can  be  greatly 
reduced  through  purchase  from  only  authorized 
resellers,  careful  examination  of  product  packaging 
and  media,  and  technology  to  notify  user  when  they  may  be  the  victim  of  counterfeit  software 


Software 

Sourcing 

•  Procurement 

Software  Development 
and  Testing 

•  Environment 

•  Personnel 

•  Software  Development 

Software 

Delivery 

•  Distribution 

•  Sustainment 


►  Cryptographic  Hashed  or  Digitally  Signed  Components  —  digitally  signed  components  or 
checksum  hashes  are  an  essential  authenticity  control  to  prove  that  components  are  genuine 


►  Notification  Technology  —  Vendors  can  leverage  technology  to  detect  certain  aspects  of  the 
product's  integrity  and  notify  the  user  if  the  software  is  deemed  to  be  counterfeit. 

►  Authentic  Verification  During  Program  Execution  —  the  practice  of  verification  when  the 
application  is  installed  on  a  computer;  each  time  the  application  runs,  the  integrity  of  the  files  is 
verified 


1  Business  Software  Alliance,  Piracy  Study,  May  12,  2009 

Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 
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Software  Delivery  Controls:  Deployment  and  Sustainment 


► 


► 


The  software  life  cycle  extends  beyond  delivery 
of  the  initial  software  vendor's  product  and  into 
the  product's  sustainment  or  maintenance 
phase. 

As  a  result,  patches  and  hot  fixes  should  be 
subject  to  the  same  software  integrity  controls 
as  the  original  code 


Software 

1  Sourcing 

u  •  Procurement 

Software  Development 
and  Testing 

•  Environment 

•  Personnel 

•  Software  Development 

Software 

Delivery 

•  Distribution 

•  Sustainment 


►  Only  authorized  service  personnel  with  ongoing  access  to  genuine  parts  and  proper 
disposal  procedures  should  be  involved  in  the  sustainment  process 


►  Secure  Configurations  —  Whenever  possible,  software  vendors  should  ship 
products  with  a  secure  configuration  being  set  as  the  default  configuration 

►  Custom  Code  Extensions  —  Integrators  must  follow  secure  development  practices 
as  they  extend  code  functionality  through  the  provided  secure  interfaces 


1  Business  Software  Alliance,  Piracy  Study,  May  12,  2009 

Source  -  SAFECode:  Software  Integrity  Controls:  An  Assurance-Based  Approach  to  Minimizing  Risks  in  the  Software  Supply  Chain 
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►  Globalization  Challenges 

►  Understanding  The  Problem 

►  Working  Towards  A  Solution 


SCRM  Stakeholders 

US  (CNCI )  has  vital  interest  in  the  global  supply  chain. 


Other  Users 


DoD 


DHS  &  IA 


PROCESS 

Commercial  ‘  v 
Industry 


SCRM  “ commercially 
acceptable  global 
standard(s)” 
must  be  derived  from 
Commercial  Industry  Best 
Practices. 


SCRM  Standardization  Requires  Public-Private  Collaborative  Effort 

Courtesy  of  Don  Davidson,  OSD  TMSN  , Chief  of  Outreach  and  Standardization 


Major  Efforts  being  executed  by  DDRE/SE 


Implementing  5200.39  and  5000.02  Program 
Protection  Policy 

-  Review/Coordination  of  PPPs  for  ACAT  I 
programs 

-  Program  protection  assessment  methodology 

-  Guidance  and  best  practice  countermeasures, 
education  and  training,  industry  outreach,  to 
assist  programs  with  CPI  identification  and 
protection 

Supply  Chain  Risk  Management 

-  Procedures,  capability  to  utilize  threat 
information  in  acquisition 

-  Commercial  standards  for  secure  components 
(ISO/IEC,  The  Open  Group) 

Horizontal  Protection  Procedures 

-  Acquisition  Security  Database  (ASDB)  oversight 
and  implementation 


Advancing  the  practice:  Systems  Security 
Engineering 

-  SERC  Research  Topic  -  “Security  Engineering” 

-  INCOSE  Working  Group  on  Systems  Security 
Engineering 

-  DoD/NSA  Criticality  Analysis  Working  Group 

DoD  Anti-Tamper  Executive  Agent 

-  Anti-Tamper  IPT,  AT  policy,  guidance  advocate 

-  Legislative  Proposal  -  Defense  Exportability 
Fund  Pilot  Program 

Countering  Counterfeits  Tiger  Team 

-  Lifecycle  strategy  to  reduce  counterfeits, 
especially  microelectronics 


Source:  Source:  September  28,2010  SwA  Forum,  DoD  Trusted  Defense  Systems,  Ms.  Kristen  Baldwin,  DDR&E/Systems  Engineering 


The  Open  Group 

Trusted  Technology  Provider  Framework  (TTPF) 


Purpose 

Identify  and  gain  consensus  on  common  processes,  techniques,  methods,  product  and  system  testing 
procedures,  and  language  to  describe  and  guide  product  development  and  supply  chain  management  practices 
that  can  mitigate  vulnerabilities  which  could  lead  to  exploitation  and  malicious  threats  to  product  integrity. 

Objectives 

•  Identify  product  assurance  practices  that  should  be  expected  from  all  commercial 
technology  vendors  based  on  the  baseline  best  practices  of  leading  trusted  commercial 
technology  suppliers 

•  Help  establish  expectations  for  global  government  and  commercial  customers  when  seeking 
to  identify  a  trusted  technology  supplier 

•  Leverage  existing  globally  recognized  information  assurance  practices  and  standards 

•  Share  with  commercial  technology  consumers  secure  manufacturing  and  trustworthy 
technology  supplier  best  practices 

•  Harmonize  language  used  to  describe  best  practices 


Source:  Source:  September  28,2010  SwA  Forum,  DoD  Trusted  Defense  Systems,  Ms.  Kristen  Baldwin,  DDR&E/Systems  Engineering 


Standards  Development  Organizations  Landscape: 


an  SCRM  Persoective 

ICT  SCRM 
Standards 
Landscape 


KEY 

International  Standards  Body 
National  Standards  Body 
Other  Organizations 


Technical  Committees/ 
Other  Standards  Bodies 


ISO,  IEC,  and  ITU 
Subcommittees 


Liaison  Relationship 
with  SC7 


Liaison  Relationship 
with  SC27 


Courtesv  of  Don  Davidson.  OSD  TMSN  .Chief  of  Outreach  and  Standardization 


CS1  ICT  SCRM  Ad  Hoc  Group 


►  Established  in  February  2009 

►  Joint  with  SC7  TAG 

►  Substantial  industry  and  government  participation 

►  Contributed  to  several  new  and  under  revision  standards 

►  Developed  consensus-based  USNB  proposal  for  ICT  Supply  Chain  Assurance  Standard 


Courtesy  of  Nadya  Bartol,  Booz  Allen  Hamilton 


ISO/IEC  JTC1  SC27  focuses  on  IT  Security  Techniques 


Working  Group  4 
Security  Controls  and 
Services 


Courtesy  of  Nadya  Bartol,  Booz  Allen  Hamilton 
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What  is  the  Problem  and  Gaps  We  Are  Trying  to  Address? 


Problem 

►  Information  and  Communication  Technology  (ICT) 
products  are  assembled,  built,  and  transported  by 
multiple  vendors  around  the  world  before  they  are 
acquired  without  the  knowledge  of  the  acquirer 

►  Abundant  opportunities  exist  for  malicious  actors  to 
tamper  with  and  sabotage  products,  ultimately 
compromising  system  integrity  and  operations 
evidenced  by  multiple  recently  publicized 
incidents  (counterfeit  hardware  sold  to  government 
agencies) 

►  Organizations  acquiring  hardware,  software,  and 
services  are  not  able  to  understand  and  manage  the 
security  risks  associated  with  the  use  of  these 
products  and  services 


Need 

►  Provide  a  common  language  for  addressing  the 
problem 

►  Provide  a  resource  that  would  help  acquirers 
articulate  requirements  to  product  and  service 
providers  and  monitor  implementation  in  a 
recognizable  manner  that  is  vetted  internationally 

-  Increase  confidence  in  acquired  products  and  services 
from  security  risk  point  of  view 

-  Create  a  common  language  to  articulate  expectations 
regarding  security  risks  associated  with  product  and 
service  acquisition 

►  Provide  a  resource  that  would  help  product  and 
service  providers  demonstrate  responsible 
practices ,  regardless  of  where  they  are  located 


Courtesy  of  Nadya  Bartol,  Booz  Allen  Hamilton 


ISO/IEC  27036:  Information  technology  -  Security  techniques  - 
Information  Security  for  Supplier  Relationships  (proposed  title) 


►  Scope:  This  international  standard  covers  information  security  in  relationships  between  acquirers  and 
suppliers  to  provide  appropriate  information  security  management  for  all  parties.  In  particular,  it  also 
includes  management  of  information  security  risks  related  to  these  relationships. 

►  The  standard  will  be  subdivided  into  the  following  parts: 

-  Part  1  -  Overview  and  Concepts 

-  Part  2  -  Common  Requirements 

-  Part  3  -  Guidelines  for  ICT  Supply  Chain 

-  Part  4  -  Guidelines  for  Outsourcing 

►  Relevant  Documents  to  be  considered 

-  Management  Systems:  ISO/IEC  27000  family;  ISO  28000,  Supply  Chain  Resiliency;  ISO/IEC  20000,  IT 
Service  Management 

-  Risk  Management:  ISO  31000,  ISO/IEC  27005,  and  ISO/IEC  16085 

-  Lifecycle  Processes  and  Practices,  software  acquisition,  and  software  assurance  ISO/IEC/IEEE  15288 
(systems),  ISO/IEC/IEEE  12207  (software),  IEEE  1062  (software  acquisition),  ISO/IEC15026  (software 
assurance) 

-  ISO  TMB  NWIP  on  Outsourcing 

Courtesy  of  Nadya  Bartol,  Booz  Allen  Hamilton 
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What’s  next? 


►  Participate  in  SC7  TAG  intersections 
through  your  SC7  TAG 

►  Participate  through  your  IEEE 
representative  to  the  SC7  TAG 

►  Participate  through  the  SwA  Working 
Groups  and  Forum 

►  Stay  Tuned  ... 


ICT  Supply  Chain  Risk  Management  (SCRM) 


Source:  September  28,2010  SwA  Forum,  Standards 
Landscape  and  ICT  SCRM  Study  Period  Update,  Nadya  Bartol, 
Booz  Allen  Hamilton 
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https://buildsecurityin.us-cert.gov/swa/proself_assm.html 


The  DHS  SwA  Processes  and  Practices  Working  Group  has  synthesized  the  contributions  of  leading 
government  and  industry  experts  into  a  set  of  high-level  goals  and  supporting  practices  (an  evolution 
of  the  SwA  community’s  Assurance  Process  Reference  Model) 

The  goals  and  practices  are  mapped  to  specific  industry  resources  providing  additional  detail  and 
real  world  implementation  and  supporting  practices 

•Assurance  Focus  for  CMMI 

•Building  Security  In  Maturity  Model 

•Open  Software  Assurance  Maturity  Model 

•CERT®  Resilience  Management  Model 

•CMMI  for  Acquisition 

•CMMI  for  Development 

•CMMI  for  Services 

•SwA  Community’s  Assurance  Process  Reference  Model  -Initial  Mappings 

•SwA  Community’s  Assurance  Process  Reference  Model  -  Self  Assessment 

•SwA  Community’s  Assurance  Process  Reference  Model  -  Mapping  to  Assurance  Models 

Other  valuable  resources  that  are  in  the  process  of  being  mapped  include 

•NIST  IR  7622:  DRAFT  Piloting  Supply  Chain  Risk  Management  Practices  for  Federal  Information  Systems 
•NDIA  System  Assurance  Guidebook 
•Microsoft  Security  Development  Lifecycle 
•SAFECode 


□  MICRO 
FOCUS 

Leading  the  Evolution  - 


r 


Micro  Focus  Applications  Value 


Profile  &  CMMI 


j 


Pete  DuPre'  &  John  Hostetler 
Micro  Focus 


Agenda 


Framing  the  Issue 
AVP  Overview 
CM  Ml  &  The  AVP 
Q&A 
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Why  did  we  develop  the  AVP  ? 


Software  project  success  continues  to  decline... 


mi  liiii 


2000 

2002 

2004 

2006 

2008 

28% 

34% 

29% 

35% 

32% 

23% 

15% 

18% 

19% 

24% 

49% 

51% 

53% 

46% 

44% 

■  Succeeded  ■  Failed  ■  Challenged 


to  2008  Project  Resolution 


2006  -  2008 
Cost  Overruns  f  7% 

Time  Overruns  f  7% 


STANDISH 

GROUP 


The  Standish  Group  International,  Inc. 
2009  CHAOS  Summary  2009 
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How  much  do  you  waste  on  rework  -  every  year? 


Number  of 
Developers 

Your  Annual 
People  Cost 

Your  Annual  Waste 

on  Rework 

30 

$3,000,000 

$1,200,000 

50 

$5,000,000 

$2,000,000 

100 

$10,000,000 

$4,000,000 

200 

$20,000,000 

$8,000,000 

300 

$30,000,000 

$12,000,000 

500 

$50,000,000 

$20,000,000 

1000 

$100,000,000 

$40,000,000 

3000 

$300,000,000 

$120,000,000 
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We've  been  talking  about  Requirements  for  a  very  long  time... 
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focus 

Leading  the  Evolution  * 


Areas 


—  kpv  Pmew 

Defect  Prevention 
Technology*  Change  M; 

PtocecsClmilHHSI 

.^teasUtvM^ 

7oigauS««’PI0CesS 
„  Orgatu^t*011  Process 
.  Training  Program 

.Integrated  Software  * 

I  .  Software  Product  Eng 

I  .  imetpoop  Cooriinat 
1  .  Peer  — 

I  *  s°oS  St*coattac«  Management 

1  Software  Quality  Assurance 

people  (and  tool«)  _ _ _ 


Focus 


Level 
Level  5 
Optimizing 


product  and  process  quality 


Level  4* 

Managed 


Engineering  processes 

otaamzahotulwppori 


Level  3 
Defined 


^^S^en,<m 

Saw 

'"Ca^^^w™600 + ,pp°  <°™9 


Level  - 

Repeatabk 


Project 


(PPQA) 


Software  Development  time  wasted  (Rework) 
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Analyst  Perspectives 


Gartner. 

Requirements  Form  the  Foundation  of  Software  Quality 

26  March  2QD9 
Thomas  E.  Murphy 

Gartner  PAS  Core  Research  GOO 165755 


Development  and  testing  teams  can't,  effectively  deliver  software  without  appropriate  requirements.  Best  practices  and  tools  can  drive  improved  quality 
and  productivity. 

Overview 

A  large  number  of  defects  ana  injected  into  software  when  the  requirements  are  collectedr  but  the  defects  aren't  detected  until  the  testing  phase. 
Involving  the  quality  assurance  {QA)  team  in  requirement  reviews  can  detect  and  remove  a  high  percentage  of  these  defectsr  improving  project 
efficiency  and  reducing  costs. 

Key  Findings 


■  Finding  and  fixing  defects  during  the  develop menfc  of  the  requirements  is  more  than  six  times  more  cost- effective  than  doing  so  during  the 
development  phase. 

i  Requirement  defects  remain  a  targe  percentage  of  defects,  and  cause  IT  versus  business  friction. _ 

*  Communication  defects  are  driven  out  by  better  requirement-eliciting  practices. 

*  Involving  the  QA  organization  in  a  sign-off  review  leads  to  early  defect  detection  and  better  testing. 

Recommendations 


*  Before  implementation  beginsr  establish  QA-d  riven  requirement  reviev^s. 

*  To  minimize  rework  costsF  invest  in  improving  the  quality  of  requirements. 

*  To  reduce  communication  errors,  use  newer  requirement-eliciting  tods  and  prototype ng  with  shoit  cycles. 


Sample  cost  savings  from  earlier  attention  to  quality 


Requirements  Design 

Cost  of 

Development 

Testing 

Production 

Resolution 
cost  for  1 00 

problem 

resolution 

lx 

2x 

lOx 

50x 

lOOx 

defects  at 
$1 00/defect 

* 

resolved 

0% 

0% 

0% 

0% 

1 00% 

"\ 

cost 

$0 

$0 

$0 

$0 

$1,000,000 

$1,000,000 

% 

resolved 

10% 

0% 

0% 

60% 

30% 

cost 

$1,000 

$0 

$0 

$300,000 

$300,000 

$601 ,000 

% 

resolved 

10% 

40% 

25% 

20% 

5% 

cost 

V 

$1,000 

$8,000 

$25,000 

$100,000 

$50,000 

$184,000 

_ _ y 

Forrester  Research,  2006 
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Why  Do  a  Value  Profile 


•  Understand  Current  Capabilities 

-  Where  do  you  stand  in  the  basic  SDLC  disciplines  ? 

-  What  progress  have  you  made  ? 

•  Compare  Capabilities  with  Industry  Norms 

-  Where  do  other  companies  stand  ? 

•  Understand  Business  and  Economic  Implications 

-  What  are  your  Critical  Business  Issues  ? 

-  What  is  the  economic  impact  of  the  CBI  gaps  ? 

-  What  is  the  value  of  closing  the  CBI  gaps  ? 

•  Determine  a  Strategy  for  Improvement 

-  What  is  the  long-term  vision? 

-  Which  improvement  steps  are  you  ready  for? 

-  Where  should  you  focus  your  efforts  in  the  short  term 
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Industry  Models 
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Enterprise  /  non-domain  specific  domain 

Lean  FDA/510K  /  spedfc 

Governance  si!lSlgma  sox  / 

EFQM 

- COBtT- — 1809900;- - 

Organizational  _ 

Infrastructure,  P‘CMM  /  'SO12207 

readiness  SCOR  |JIL  SWEBOK 

(incl  biz  practices,  engr  practices, 

change/improvement  practices) _ OQllVl _ PSM _ 

i 

6S/DMAIC  (DEAL 

Tactical 

{procedural  -  both  for  6S/DFSS  TSR 

improvement  tasks  and  Agile 

for  engrg  tasks)  /  RUP 
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What  are  we  doing  in  an  AVP  ? 
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1)  Business  Issues 


FtegliliftignS  &  Compliance 
Customer  Sat  refaction 
Feat  iu  re  I  m  pie  mental  bn 
Applies!  ton  Development  Costs 
Time  to  Market 


2)  Financials 


Application  Development  Budget 


Development  Staff 

5D%  ' 

Project  Management 

10% 

51,500,000 


3)  Processes 


Test  Environment  Preparation 

Test  Env  ronments 

Ded  cated  test  env  ronments  are  made  ava  ah  e  n  a  t  ime  y  manner  to  cover 
d  ffe  r  e  nt  d  e  p  o  v  m  e  n  t  e  n v  t  o  n  m  e  nts 

Privacy 

A  data  popu  at  ngthe  ^est  env  ronments  fo  ow  Corporate  pc  c'sj  for  data 
pr'vacy,  SOX.  H  FAA,  ate. 

O-e 

4)  Value  of  Improvement 


Sources  of  Business  Value 

Lower  Costs 

$252,332 

Higher  Quality 

$87,293 

Faster  Time  to  Market 

$50,194 

Better  Business  Alignment 

$100,660 

Total 

$ 490,479 

ii 


AVP  Content 
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Process  Analysis 


Capability  Profile 


Repeatability 


Precision  Visibility/Controlled  Optimized 


Project  Management 

Foundational  Phases 

Phase  Management 

Risk  Management 

Governance 

40% 

45Vo\ 

;  343fi 

20^ 

II  Quality 

Qualify  Control 

Quality  Assurance 

Qualify  Management 

Quality  Governance 

■  10% 

im\ 

■  13% 

K  T% 

7% 

Change  &  Configuration 
Management 

Essentials 

Planning  and  Change 
Management 

Activity/Asset 

Management 

Governance 

— 

Uj 

m 

19% 

19% 

1% 

Requirements 

Document  Focused 

Stahility  Focused 

Alignment  Focused 

Total  RDM  Focused 

42% 

1Q% 

9% 

Current  Capability? 


Industry  Comparison 


Project  Management 

100>r 


Change  ft 
Configuration 
Management 


■♦—Some  Company 
-Q—  IT  Average 


Where  do  you  stand  relative 
to  industry? 


Requirements 
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Projected  Value  of  Improved  Capability 


Cumulative  Annual  Business  Value 


□Value  Opportunity  ($1,331541) 


Improvement  Value 

Phase  1 

$490,473 

Phase  2 

$257,942 

Phase  3 

$305,922 

Phase  4 

$95,399 

Phase  5 

$G2.14G 

Phase  G 

$130,052 

hGtiJl 

Phase  One 


Sources  of  Business  Value 

Lower  Costs 

51.45% 

$29  2,332 

Higher  Quality 

17.30% 

$87,293 

FastE  r  Ti  m  e  to  M  a  rket 

IQ. 23% 

$53,194 

0  ettE  r  0  u  i\  n  ess  A 1  i  gn  m  e  nt 

20.52% 

$130,560 

fOL-ffl 

109.90% 

USO.  473 

What  is  the  potential  value  of 
a  Phase  1  improvement 
program? 
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AVP  Supports: 

CMMI  Infrastructure  Institutionalization 


•  Institutionalization  means  that  the  process  is  ingrained  in  the  way  the 
work  is  performed:  "That's  the  way  we  do  things  around  here." 

•  The  organization  builds  an  infrastructure  that  contains  effective,  usable, 
and  consistently  applied  processes  (e.g.,  GP  2.3) 


GP  2.3:  Provide  Resources 

Provide  adequate  resources  for  performing  the  <x>  process,  developing  the 
work  products,  and  providing  the  services  of  the  process. 
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The  "Value"  of  AVP  during  CMMI  Appraisa 


With  AVP,  a  detailed  analysis  on  capability  is  created  to  help  support  "GP  2.3  -  Provide  Resources"  from  a 
holistic  view  in  knowing  what  are  your  gaps/strengths  and  what  technology  improvements  could  be  helpful  in 
supporting  the  specific  process  areas. 


/ 

7 

GG1:  Achieve  Specific  Goals 

GP  1.1:  Perform  Specific  Practices 

/ 

GG2:  Institutionalize  a 

GP  2.1 

Establish  an  Organizational  Policy 

Managed  Process 

GP  2.2 

HP  ?  ? 

Plan  the  Process 

Or  L. .O 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Manage  Configurations 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Management 

/ 

GG3:  Institutionalize 

GP  3.1 

Establish  a  Defined  Process 

a  Defined  Process 

GP  3.2 

Collect  Improvement  Information 

/ 

GG4:  Institutionalize 

GP  4.1 

Establish  Quantitative  Objectives  for  the  Process 

a  Quantitatively  Managed 

GP  4.2 

Stabilize  Subprocess  Performance 

Process 

/ 

GG5:  Institutionalize 

GP  5.1: 

Ensure  Continuous  Process  Improvement 

an  Optimizing 

GP  5.2: 

Correct  Root  Causes  of  Problems 

Process 

7 
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AVP  Supports  Identifying  Possible  Innovations 
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The  Relationship  of  AVP<^>OID 

For  those  organizations  targeting  to  define  and  deploy  high  maturity  practices  within  their 
organization,  the  AVP  enables  an  even  stronger  implementation  of  Organizational  Innovation  and 
Deployment  (OID)  practices  especially  surrounding  the  "active"  search  outside  the  organization's 
IT  domain.  Basically,  OID's  specific  practice  (SP)  1.2  "Identify  and  Analyze  Innovations"  is  fully 
optimized  to  increase  the  organization's  quality  and  process  performance. 


ALM  Maturity  Profile 


What  Progress  Have  You  Made? 


How  Do  You  Compare  With  Others? 


What  Are  Your  CBI's 


How  Could  You  Close  The  CBI  Gaps? 


What  Disciplines  Were  Profiled? 
What  Development  Approach? 


What  Is  The  Economic  Impact? 


Economic  Impact  of  CBI  Gaps 


Potential  Economic  Impact  of  Higher  Matu 

Quality  4  181090  -  4327.346* 


Example:  The  Economic  Opportunity 


•  Organization  with  $7M  annual  development  budget 

•  $850k  projected  annual  benefit 

•  Just  by  optimizing  requirements  definition  &  management 


Projected  Benefits 


$0  $100,000  $200,000  $100,000  $400,000  $500,000  $600,000  $700,000  $600,000  $900,000 


■  Lower  Costs  □  Higher  Quality  □  Faster  Tirr-e  to  Market  □  Better  Business  Alignment 
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Q&A 
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Carnegie Mellon 

Software  Engineering  Institute  l 


Pittsburgh,  PA  15213-3890 


O  C  K  H  E  E  D 


CMMI  VI. 3 
High  Maturity  Panel 

WE  THOUGHT  BUT  NOW 

WE  KNOW 


M.  Lynn  Penn 

Lockheed  Martin  Information  Systems  &  Global  Services 

Rawdon  Young 

Software  Engineering  Institute 


Alice  Perry 

Raytheon 


Page  1 


Carnegie  Md  Ion 

Software  Engineering  Institute 


LOCKHEED 


MARTIN 


Presentation  Agenda 


TWO  PARTS 

1.  WE  THOUGHT 

Background  -  how  we  got  here 


2.  WE  KNOW 

CMMI  VI. 3  High  Maturity  Process  Areas 


Page  2 


Carnegie  Md  Ion 

Software  Engineering  Institute 


LOCKHEED 


MARTIN 


The  Challenge 


High  Maturity  CMMI  VI  .3 
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Carnegie  Md  Ion 

Software  Engineering  Institute 


LOCKHEED 


MARTIN 


High  Maturity  Starting  Point 


The  High  Maturity  Team  is  composed  of  veterans  of  the 
VI. 2A  workshop  -  note  the  majority  are  from  high 
maturity  organizations  in  industry. 

•  Lynn  Penn  (LMCO)  -  Lead 

•  Dan  Bennett  (AF) 

•  Will  Hayes  (SEI) 

•  Rick  Hefner  (NG) 

•  Jim  Kubeck  (LMCO) 

•  Alice  Parry  (Raytheon) 

•  Kathy  Smith  (HP) 

•  Rusty  Young  (SEI) 

Use  of  the  high  maturity  redline  of  CMMI  for  Development 
and  change  requests  submitted  against  it  has  added 
another  layer  of  complexity  to  the  analysis  of  change 
requests. 

•  Assumption  -  CMMI  Dev  VI  .2  is  baseline 

•  Assumption  -  CMMI  Dev  VI  .2A  (redlines)  is  a  CR 
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High  Maturity  Issues 


•Terminology  Confusion 
•Requirements  implied  versus  explicit 

•Explanations  not  central  or  consistent 

•  Model/  Audit  Criteria/  Presentations  (Healthy  Ingredients)/ 
UCHMP 

•Perceptions 

•  Customers  -  ML  5  is  expensive  -  no  better  than  3 

•  Industry  -  ML  5  is  NOT  RIGHT  for  every  business 

•High  Maturity  in  ALL  constellations 

•  Examples  are  focused  on  Development 
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Terminology  Confusion 


•Common  Cause 

-  Statistical  versus  QuantitativeTechniques 
•Process  Models  and  Process  Modeling 
•Business  Objectives 
•Subprocesses 
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Two  issues  really  one 


•  Goal: 

-  Sunset  -  OTHER  explanations 

-  Incorporate  Healthy  ingredients  as  appropriate  - 
goals/  practices 

-  Audit  Criteria  -  Audit  to  Model  and  MDD  - 
include  what  is  necessary 
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Perceptions 


Individual  based...  However  recommended 
HIGH  MATURITY  RESTRUCTURING 

•  Insufficient  link  between  process  improvement, 
business  objectives,  and  performance 

•  Clarify  distinction  between  ML4  and  ML5 

•  Eliminate  GG4  and  GG5 

•  CMMI  VI  .3  Webnar  Suggestions  (not  covered 
elsewhere) 

Clarify  role  of  OID  and  CAR 

Make  CAR  more  relevant/  clarification  in  role 
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High  Maturity  Pre-Review 

High  Maturity  Review  at  SEPG 

•  Invited  individuals  which  include  HMLA/  Potential  HM 
Pilot  companies  to  a  special  session  at  the  NA  SEPG 

-  Rusty  and  Lynn  presented  the  HM  Team  intentions 
and  introduction  to  the  HM  PAs  (including  the  new 
OPM  PA) 

-  Provided  these  same  individuals  with  the  redlines  for 
HM  including  a  high  level  OPM  (GLOP) 

-  Asked  for  the  following  actions: 

-  Within  1  week  provide  HMT  with  a  heads  up  or 
down  on  changes 

-  Within  2  weeks  provide  HMT  with  redlines  to  the 
redlines 
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High  Maturity  SEPG  Results 

•Approximately  150  individuals  were  invited 

•Approximately  45  individuals  attended 

•Received  feedback  at  conference  -  positive  on  intentions 

and  direction  we  were  pursuing 

•Received  heads  up/  down  feedback  from  6  individuals 

-  One  of  these  was  not  at  presentation  so  took  many 
emails  to  explain 

-  The  feedback  was  more  questions  -  not  in  keeping 
with  initial  SEPG  positive  feedback 

-  Four  of  these  individuals  have  also  provided  redlines 
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High  Maturity  Team  Consensus 

•Team  members  that  attended  SEPG  were  positive  and 
very  upbeat  about  questions  and  follow  up 
•Observations  after  feedback 

-  Discouraged  that  so  few  responded 

-  A  few  comments  showed  a  mis-understanding  of 
existing  HM  and  an  inability  to  grasp  anything 
different 

•Team  Position 

-  Clean  up  existing  PAs  (some  relevant  comments) 

-  Complete  OPM  generation 

-  Proceed  to  “Open  Team”  review 
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Improvements 


Customer 
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High  Maturity  PAs  Relationships  1 


Organizational  Process  Performance  (OPP) 

•  Clarified  relationship  between  OPP  and  rest  of  high  maturity 
process  areas 

•  Clarified  that  process  performance  baselines  and  models  can 
be  created  and  used  in  at  levels  and  not  just  the 
organizational  level 

Quantitative  Project  Management  (QPM) 

•  Restructured  to  a  prepare  for  quantitative  management  and 
quantitatively  manage  the  project  or  work. 

•  Emphasized  the  use  of  statistical  and  other  quantitative 
techniques 

•  Emphasized  that  quantitative  management  covered  managing 
subprocesses  through  the  project  levels  (from  the  micro 
through  the  macro  levels) 
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High  Maturity  PAs  Relationships  1 


rity 


can 


Ojquantitative  management 

•  Managing  a  project  or  work  group  using  statistical  and  other 
quantitative  techniques  to  build  an  understanding  of  the 
Derformance  or  predicted  performance  of  processes  in 

•  comparison  to  the  project’s  or  work  group’s  quality  and  process 
Derformance  objectives,  and  identifying  corrective  action  that 
nay  need  to  be  taken.  (See  also  “statistical  techniques”) 

Statistical  techniques  used  in  quantitative  management  include  analysis,  creation,  or 
use  of  process  performance  models,  analysis,  creation,  or  use  of  process 
Q  Derformance  baselines;  use  of  control  charts;  analysis  of  variance,  regression 
analysis;  and  use  of  confidence  intervals  or  prediction  intervals,  sensitivity  analysis, 

•  simulations,  and  tests  of  hypotheses. 

quantitativeiyrTfcri^^Ihe  project  or  work. 

•  Emphasized  that  quantitative  management  covered  managing 
subprocesses  through  the  project  levels  (from  the  micro 
through  the  macro  levels) 

•  Emphasized  the  use  of  statistical  and  other  quantitative 
techniques 


nd 
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High  Maturity  PAs  Relationships  1 


statistical  and  other  quantitative  techniques 

Analytic  techniques  that  enable  accomplishing  an  activity  by  quantifying 
Darameters  of  the  task  (e.g.,  inputs,  size,  effort,  and  performance).  (See 
also  “statistical  techniques”  and  “quantitative  management.”) 

This  term  is  used  in  the  high  maturity  process  areas  where  the  use  of 
statistical  and  other  quantitative  techniques  to  improve  understanding  of 
Droject,  work,  and  organizational  processes  is  described. 

Examples  of  non-statistical  quantitative  techniques  include  trend  analysis, 
•un  charts,  Pareto  analysis,  bar  charts,  radar  charts,  and  data  averaging. 
The  reason  for  using  the  compound  term  “statistical  and  other  quantitative 
techniques”  in  CMMI  is  to  acknowledge  that  while  statistical  techniques 
are  expected,  other  quantitative  techniques  can  also  be  used  effectively 


through  the 

•  Emphasized  the  use  of  statistical  and  other  quantitative 
techniques 
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High  Maturity  PA  Relationships  2 

Causal  Analysis  and  Resolution  (CAR) 

•  Made  it  clearer  when  to  use  statistical  and  other  quantitative  techniques. 

•  Clarified  use  by  projects  AND  organizations  to  perform  causal  analysis 
and  resolution  on  selected  outcomes. 

•  Updated  to  include  positive  and  negative  outcomes  -  not  just  defects. 

•  Modified  outputs  from  CAR  to  include  Improvement  Proposals  to  feed 
process  improvements  to  the  organization’s  set  of  standard  processes 
(OPM). 

Organizational  Performance  Management  (OPM) 

•  Focused  the  Process  Area  on  managing  business  performance  to 
achieve  quality  and  process  performance  objectives. 

•  Added  a  Specific  Goal  that  requires  organizations  to  use  measures, 
process-performance  baselines  and  models  from  OPP  to  understand 
process  performance,  target  areas  for  continuing  improvement,  and 
evaluate  the  impact  of  proposed  improvements. 

•  Made  it  clear  that  statistical  and  other  quantitative  techniques  are  used  to 
evaluate  and  select  improvement  proposals  and  to  evaluate  whether  the 
improvement  achieved  expected  performance  improvement. 
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Organizational  Process  Performance 


Changes: 

•  Restructured  OPP  moving  “Establish  Quality  and  Process 
Performance  Objectives  to  SP  1.1  for  emphasis. 

•  Revised  SP  1 .4  to  include  process  performance  analysis  and 
assessment  of  subprocess  stability. 

•  Revised  SP  1 .5  to  clarify  process  performance  models  are 
used  throughout  the  development  lifecycle  toward  achieving 
quality  and  process  performance  objectives. 

•  Clarified  that  not  all  process  performance  baselines  and 
models  must  be  created  at  the  organization  level.  Projects 
can  follow  OPP  practices  to  create  process  performance 
baselines  and  models,  when  appropriate. 

•  Clarified  the  relationship  of  OPP  to  other  high  maturity 
process  areas. 

•  Emphasized  traceability  to  business  objectives  through 
modifications  to  SP1 .1  and  SP1 .2 
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Quantitative  Project  Management 


Changes: 

•  Restructured  QPM  so  that  SGI  focuses  on  preparation  and 
SG2  focuses  on  managing  the  project. 

•  Broadened  the  focus  on  using  statistical  techniques  from 
individual  selected  subprocesses  to  cover  multiple  levels  from 
the  individual  subprocesses  to  the  entire  project. 

•  Added  guidance  about  using  process  performance  baselines 
and  process  performance  models. 

•  Defined  quantitative  management  in  the  glossary  to  include 
statistical  techniques  and  used  that  definition  for  use  of  the 
terms  throughout  QPM. 

•  Modified  the  practice  to  remove  the  emphasis  on  applying 
statistical  methods  to  understand  variation  to  reduce  the  over¬ 
emphasis  on  control  charts. 

•  Added  new  practices  about  managing  performance  and 
performing  root  cause  analysis. 
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Causal  Analysis  and  Resolution 


Changes: 

•  Used  “outcomes”  to  include  positive  outcomes  instead 
of  only  “defects  and  problems.” 

•  Added  examples  for  service  organizations  and  for 
selecting  outcomes  for  analysis. 

•  Added  subpractices  in  SP  1.1  for  defining  the  problem, 
and  in  SP  2.2  for  following  up  when  expected  results 
were  not  realized. 

•  Added  more  information  about  how  PPMs  can  be  used. 

•  Added  informative  material  addressing  more  proactive 
defect  prevention. 
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Organizational  Performance 
Management 

Changes: 

•  Expanded  the  former  OID  PA  to  include  performance 
management  and  called  it  Organizational  Performance 
Management  (OPM)  to  emphasize  focus  on  performance  of  the 
organizational  processes  as  they  relate  to  business  objectives. 

•  Defined  a  new  goal  about  managing  business  performance  using 
statistical  and  other  quantitative  techniques. 

•  Clarified  that  improvements  selected  for  possible  implementation 
can  be  validated  in  different  ways,  piloting  is  not  the  only  option. 

•  More  explicitly  described  the  use  of  process  performance  models. 

•  Provided  more  information  about  how  improvements  can  be 
selected  for  deployment. 

•  Changed  references  from  “process  and  technology 
improvements”  to  “improvements”,  with  an  explanation  that 
improvements  include  both. 
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Scott  Foster 
Dorna  Witkowski 


Agenda 


*  Background 

*  Purpose  of  Pilot 

*  Streamlined  Structure  of  Pilot 

*  Summary  of  Results 
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X 


Background  -  IS&GS-Defense 

•  Lockheed  Martin  IS&GS-Defense 

-  Employs  over  12,000  people  at  more  than  200  sites 
encompassing  over  400  programs  worldwide 

-  Is  principally  engaged  in  the  design,  development, 
operation  and  sustainment  of  systems  and 
solutions  that  help  our  defense  customers  achieve 
their  missions 

-  Headquarters:  Valiev  Forge,  PA 

-  Customer  Basel  The  majority  of  IS&GS-Defense’s 
business  is  with  the  U.S.  Department  of  Defense, 
other  U.S.  government  customers  and  the  military 
services  of  international  governments. 
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Background  -  IS&GS-Defense 

•  In  January  2009  IS&GS-Defense  achieved  a  CMMI  Maturity  Level  5  based 
on  the  Staged  CMMI  Dev  VI. 2  +  IPPD 

-  Organization  and  3  large  programs  assessed: 

it 

•  Advanced  Extremely  High  Frequency  (AEHF)  Mission  Control  Segment 
(MCS)  -  develops  and  deploys  the  MCS  which  will  consist  of  upgrades  and 
replacement  of  the  existing  fielded  MILSTAR  equipment  and  software  to 
support  the  combined  MILSTAR  and  AEHF  constellation  of  satellites. 

-  Size:  348  People,  Development 

-  Locations:  King  of  Prussia,  PA;  Denver,  CO;  San  Jose,  CA;  Rockville,  MD 

ic 

•  Space  Based  Infrared  System  (SBIRS)  High,  Ground  Segment  - 
develops  software  to  replace  the  ground  segments  of  the  US  satellite 
Early  Warning  System. 

-  Size:  368  People,  Development 

-  Location:  Boulder,  CO;  Sunnyvale,  CA 

•  Integrated  Space  Command  and  Control  (ISC2)  program  -  provides  a 
powerful,  yet  flexible  platform  for  current  and  emerging  space,  air,  and 
missile  defense  missions. 

-  Size:  388  People,  Development 

-  Location:  Colorado  Springs,  CO  *  Also  participated  in  VI  .3  Pilot 


Background  -  High  Maturity 
Process  Areas  of  VI  .3 

•  Planning  for  the  pilot  was  initiated  on  the  March  2010  GLOP  High  Maturity  Redline 

-  Causal  Analysis  &  Resolution  (CAR)  updated 

•  Change  from  “Defects  and  Problems”  to  “Outcomes”  (good  or  bad) 

-  Organizational  Innovation  &  Deployment  (OID)  updated 

•  Differentiation  between  incremental  and  innovative  improvements 

-  Organizational  Performance  Management  (OPM)  was  added 

•  New  process  area  to  manage  the  process  performance  of  the  organization 

-  Organizational  Process  Performance  (OPP)  updated 

•  Change  to  order  of  specific  practices  to  help  understanding  of  intent 

•  Rewrite  of  selection  of  processes  and  sub-processes 

-  Quantitative  Project  Management  (QPM) 

•  Modification  of  order  of  specific  practices  to  help  understanding  of  intent 

•  Clarification  of  analysis  of  project  performance 

•  Used  this  version  as  the  baseline  for  the  pilot 

-  Did  not  change  direction  even  when  OPM  was  combined  with  OID  to  form  a  combined 
OPM 


The  assumption  was  that  VI. 3  updates  did  not  levy  new 
requirements,  only  reinforced  the  intent  of  the  high  maturity 

concepts 
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Purpose  of  Pilot 


•  Provide  feedback  to  the  Software  Engineering  Institute  (SEI)  on 
the  updated  language  within  the  model 

•  Ensure  the  company  and  its  programs  were  consistently 
performing  at  a  Maturity  Level  5 

•  Identify  any  potential  disconnects  between  the  company’s  plans 
and  processes  and  the  updated  model 

-  Ensure  that  the  organization  understood  and  was  consistent 
with  the  new  wording 

-  Help  new  leaders  in  process  positions  feel  more  comfortable 
with  the  plans  for  a  re-appraisal 

•  Piloted  new  PHD  tool  developed  by  the  organization 
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Streamlined  Structure  of  Pilot 


x 


•  Chose  a  Class  B  method 

•  Only  assessed  High  Maturity  Process  Areas 

•  Used  discovery  mode  for  QPM  and  CAR 

-  Programs  provided  easily  available  evidence  for  review  (standard  business 
rhythm  work  products) 

-  No  program  preparation  time  required 

-  Used  artifacts  from  programs  that  were  focus  programs  in  the  2009  SCAMPI  A 

•  Required  PHDS  for  specific  practices  of  organizational  process 
areas  only 

-  OPP,  OID,  OPM  ^ 

-  No  objective  evidence  for  generic  practices  ' 


Used  an  experienced  appraisal  team 
-  Both  previously  on  SCAMPI  A  for  the  organization 
Conducted  interviews  virtually 


Summary  of  Results  -  Feedback 
to  the  SEI 


•  Full  agreement  with  eliminating  GG4  and  GG5 

•  Full  agreement  on  greater  emphasis  on  Process  Performance  Baselines 
and  Models 

-  Recommended  greater  emphasis  in  the  informative  material  for  ML5 
PAs 

•  Partial  agreement  on  adding  a  new  OPM  process  area 

-  Y2  of  team  felt  it  wasn’t  needed  and  didn’t  add  real  value 

-  V2  of  team  felt  it  helped  tie  the  relationship  between  OID  and  OPP 

•  Full  agreement  with  including  positive  as  well  as  negative  outcomes  for 
CAR 

•  Mild  agreement  with  de-emphasizing  “common  cause”  as  a  differentiator 
between  ML4  and  ML5 

-  Most  people  did  not  have  an  opinion 

•  Agreement  on  broadening  “statistical  management” 

•  Agreement  on  restructuring  QPM 
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Summary  of  Results  -  Performance  of 
Organization  against  the  VI  .3  PAs  (1) 


*  Results  were  provided  as  risks: 

*  Low  risk  of  meeting  CMMI  goals  -  on  the  right  ^ 

track,  but  still  work  to  do  ' 

*  Medium  risk  of  meeting  CMMI  goals  -  actions  need 
to  be  taken  to  stay  on  track 

*  High  risk  of  meeting  CMMI  goals  -  offtrack;  need  to 
re-assess  program  or  organizational  activity 

*  Additional  meetings  held  after  the  Final  Briefing  to 
explain  results  in  detail 


Summary  of  Results  -  Performance  of 
Organization  against  the  VI  .3  PAs  (2) 


General  Strengths: 

-  Management  support  of  the  pilot 

•  Both  in  concept,  time  of  participants  and  follow-up  after  the 
results 


-  Organization  and  Programs  have  Goal-to-Measure  Mappings 
that  show  the  threads  from  high  level  business  goals  to  actual 
measurements  used 


•  General  Weaknesses: 


-  Difficulty  in  articulating  how  the  term  “Quality  and  Process 
Performance  Objectives”  related  to  standard  activities 

-  Participants  were  not  in  “appraisal”  mode 

•  Even  though  most  had  been  in  appraisals  before 

•  Had  to  be  reminded  not  to  assume  the  appraisers  knew 
how  they  operated 
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Summary  of  Results  -  Performance  of 
Organization  against  the  VI  .3  PAs  (3) 


*  Program  Process  Areas  were  low  risk 

-  Evidence  easily  available  to  show  compliance  with  QPM  and 
CAR 

-  Programs  were  being  managed  quantitatively  and 
improvements  were  tied  to  program  goals 

-  But  program  personnel  needed  to  be  able  to  better  verbalize 
threads  and  quantitative  results 

•  OPM  was  low  risk 

-  Business  needs  and  goals  were  established 

-  Improvements  were  based  on  goals 

-  Needed  to  articulate  the  “Quality  and  Process  Performance 
Objectives” 


Updates  to  QPM  and  CAR  had  no  impact  on  programs 
OPM  turned  out  to  be  a  “no  brainer” 


Summary  of  Results  -  Performance  of 
Organization  against  the  VI  .3  PAs  (4) 


OPP  was  medium  risk 

-  New  organizational  participants  needed  to  learn  to  “tell  the 


story” 


•  To  tie  together  the  various  aspects  of  high  maturity 

-  This  observation  was  driven  by  the  model’s  emphasis  on 
traceability  to  business  objectives 

-  New  order  of  OPP  did  not  affect  results 
*  OID  was  medium  risk 

-  Business  needs  and  goals  were  established 

-  More  emphasis  on  quantitative  results  in  model  will  di '  e 


emphasis  on  quantitative  results  in  the  evidence 
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Summary  of  Results  -  PHD  Tool 


•  PHD  tool  used  to  review  evidence  for  Organizational  PAs 

•  Evidence  Review  Feedback 

-  Allow  review  comments  in  the  tool 

-  Allow  partial  satisfaction  without  failing  the  record/practice 

-  Allow  information  requests  without  failing  record/practice 

-  Show  all  evidence  for  a  PA  at  a  glance 

•  Evidence  Entry  Feedback 

-  Allow  easy  transfer  between  Direct  and  Indirect 

-  Show  what  evidence  has  been  submitted  for  entire  Practice 


x 


1 


All  of  the  feedback  has  been  addressed  in  the  latest  version 
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Summary  of  Results  -  PHD  Tool 
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PHD  Owner  Interface 


Retrieve  PUDs  |  Setect  Project  ISC2  Evidence 


|  Home  |  [  Excel  Report  Go  To  Record  Contact  Us  Help 


MID 


PA 


CAR  -  Causal  Analysis  and  Resolution 


□  Show  all  PllDs  with  ps*-dirg  actiors  [All  prcjra^  PA= 


Evidence 

ISC2  CAR  SP  1.2 

Perform  causal  analysis  of  selected  defects  and  other  problems  and  propose 
actions  to  address  them. 

Practice  Helo 
History  and  Corrections 

|  Rationale:  Discusses  doing  a  Causal  Analysis  foiloiving  an  inspection. 

Title 

Description 

Hyperlink 

PCB  Minutes  from 
06/26/2008 

PCB  minutes  sumbitted  to  demonstrate  the  actions 
taken  by  the  PCB  as  they  realte  to  the  CA  presented 
to  them.  This  new  record  submitted  at  the  request  of  1 
the  auditors 

.♦CB  Minutes. doc# 

j|status:  Reviewed 

Open  | 

PA 


Practice!  Status  Select 
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0 
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SP 
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CAR 

GP 
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GP 
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Reviewed  | 

CAR 

GP 
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Reviewed  | 

CAR 

GP 

2.6 

Audited 

CAR 

GP 

2.7  ' 

Reviewed 

CAR 

GP 

2.8 

Submitted  1 

tSC2  CAR  SP  2,1 

implement  the  selected  action  proposals  that  were  developed  in  causal  analysis . 


Rationale:  The  just  do  it  section  assigns  actions  to  be  taken  for  the  quick  fixes 


Practice  Help 
History  and  Corrections1 


Title 


Description 
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Summary  of  Results  -  PHD  Tool 


PUP  Owner  Interface  -  Wind 


Action  Entry  -  Windows  Internet  Explorer 


X 


Bl§, 


ji  T  g|_,  http://www.bld.isgs.lrmc 

A  ■*  IS1 


)  I5&.G5  Home  -  PA55 


ACCT02\TO5tensd  Auditor  Mode 
teimi;  a  dim  in 


AC  V  \  0  2  \f  osiers  d  Au  ditor  Mode 
teimr:  adim'n 


Action  Entry 


24761 

P  ease  rprov'de  Root  Cuase  Ana  ys  s  based  on  exceed'ng  contro  n  ts 


Retrieve  PIIDs 


Sefect 


MID 


PA 

Practice 

Status 

*- 

CAR 

SP 

1.1 

Reviewed 

0 

CAR 

SP 

1.2 

New 

0 

l  CAR 

SP 

2.1 

New 

0 

CAR 

SP 

2.2 

New 

0 

CAR 

SP 

2.3 

New 

0 

i  CAR 

GP 

2.1 

New 

□ 

CAR  | 

GP 

2.2 

New 

□ 

CAR 

GP 

2.3 

New 

□ 

CAR 

GP 

2.4 

New 

□ 

CAR 

GP 

2.5 

New 

□ 

CAR 

GP 

2.6 

New 

□ 

CAR 

GP 

2.7 

New 

□ 

CAR 

GP 

2.8 

New 

O 

CAR 

GP 

2.9 

New 

□ 

.-  /VO 

n 

jLin 

M  j-..  .  . 

n 

Issue  and 
Recommendation 


■ 

Title 

Description 

Hyperlink 

Status:  New 

[  Open  | 

V  ] 

1 

i v 

_ 


Summary 


*  Pilot  helped  organization  do  a  “mid-course”  correction 

-  Helped  to  understand  emphasis  in  the  next  version 
of  the  model 

-  Generated  actions  to  ensure  high  maturity  activities 
remain  focused 

*  Pilot  helped  establish  a  usable  and  flexible  PHD  tool  for 
future  appraisals 

*  SCAMPI  B  on  all  organizational  process  areas 
scheduled  for  January  201 1  with  VI  .3 
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Comparing  Scrum  And  CMMI 

How  Can  They  Work  Together 


Neil  Potter 


The  Process  Group 
help@processgroup.com 
www.processgroup.com 
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*  Definition  of  Scrum 

*  Agile  Principles 

*  Definition  of  CMMI 

*  Similarities  and  Differences 

*  CMMI  and  Scrum  Mapping 

*  How  About  Other  Components  of  Level  2? 

*  How  About  Level  3? 

*  Summary 


Full  comparison  at:  http://www.processgroup.com/pgpostmar09.pdf 
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Definition  of  Scrum 


•  Scrum  is  a  pre-defined  development  lifecycle  based  on  Agile 
principles. 
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2-4  week  Sprint 


2-4  week  Sprint 
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Agile  Principles -i 

f  Our  highest  priority  is  to  satisfy  the  customer  through  early  and 
continuous  delivery  of  valuable  software. 

*  Welcome  changing  requirements,  even  late  in  development.  Agile 
processes  harness  change  for  the  customer's  competitive 
advantage. 

^  Deliver  working  software  frequently,  from  a  couple  of  weeks  to  a 
couple  of  months,  with  a  preference  to  the  shorter  timescale. 

»  Business  people  and  developers  must  work  together  daily 
throughout  the  project. 

}  Build  projects  around  motivated  individuals.  Give  them  the 
environment  and  support  they  need,  and  trust  them  to  get  the  job 
done. 


CMMI  compatible 


Source:  http://agilemanifesto.org/ 
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Agile  Principles -2 

The  most  efficient  and  effective  method  of  conveying  information 
to  and  within  a  development  team  is  face-to-face  conversation. 

Working  software  is  the  primary  measure  of  progress. 

Agile  processes  promote  sustainable  development.  The  sponsors, 
developers,  and  users  should  be  able  to  maintain  a  constant  pace 
indefinitely. 

Continuous  attention  to  technical  excellence  and  good  design 
enhances  agility. 

Simplicity  -  the  art  of  maximizing  the  amount  of  work  not  done-is 
essential. 

The  best  architectures,  requirements,  and  designs  emerge  from 
self-organizing  teams. 

At  regular  intervals,  the  team  reflects  on  how  to  become  more 
effective,  then  tunes  and  adjusts  its  behavior  accordingly. 
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Definition  of  CMMI 


vl 


5  Optimizing 

Continuous 

Process 

Improvement 

Organizational  Innovation  and  Deployment 

Causal  Analysis  and  Resolution 

4  Quantitatively 
Managed 

Quantitative 

Management 

Organizational  Process  Performance 

Quantitative  Project  Management 

3  Defined 

Process 

Standardization 

Requirements  Development 

Technical  Solution 

Product  Integration 

Verification 

Validation 

Organizational  Process  Focus 

Organizational  Process  Definition  +IPPD 

Organizational  Training 

Integrated  Project  Management  +IPPD 

Risk  Management 

Decision  Analysis  and  Resolution 

2  Managed 

Basic 

Project 

Management 

Requirements  Management 

Project  Planning 

Project  Monitoring  and  Control 

Supplier  Agreement  Management 

Measurement  and  Analysis 

Process  and  Product  Quality  Assurance 

Configuration  Management 

1  Initial 

Model  Source:  http://www.sei.cmu.edu/cmmi/tools/ 
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.2 

CMMI  is  a  collection 
of  practices  that  an 
organization 
(software,  hardware 
and  IT)  can  adopt  to 
improve  its 
performance. 

Maturity  Level  2 
Process  Areas  focus 
on  change  and 
project  management. 

Maturity  Level  3 
focuses  on 
engineering  skills, 
advanced  project 
management  and 
organizational 
learning. 


Version  1.6  6 


Visibility  Into  the  Process 

Level  1 


THE 

PRC 

GRC 


•  Process  is  an  amorphous  entity 

•  Visibility  into  the  project’s  process  is  limited 

•  Difficult  to  establish  the  status  of  the  project’s  progress 
and  activities 
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Visibility  Into  the  Process 

Level  2 


IN  | - 1 

- ►  - ► 

- a 

*  Customer  requirements  and  work  products  are  controlled 

*  Basic  project  management  practices  have  been  established 

*  Management  controls  allow  visibility  into  the  project  on 
defined  occasions 

*  Management  reacts  to  problems  as  they  occur 


Major  process  phases 
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Level  3 
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*  Tasks  in  the  project’s  defined  process  are  visible 

*  Accurate  and  rapid  status  updates  are  available 

*  Management  proactively  prepares  for  risks  that  may  arise 


©  Copyright  2004-2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  1.6 


9 


THE 

PRC 

GRC 


Similarities  and  Differences 


In  Scrum? 


No 

Some  requirements 
Some  design 
Coding 
Some  test 
Some  lessons  learned 

Most  Requirements  Management 
Most  Project  Planning 
Most  Project  Monitoring/Control 
Most  Measurement  Analysis  (effort 
and  progress) 


Level  3  coverage  - 
very  dependent  on 
how  YOU  define 
the  phases 


©  Copyright  2004-2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  1.6 


10 


THE 

PRC 

GRC 


CMMI  and  Scrum  Mapping 
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REQM 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Develop  an  understanding  with 
the  requirements  providers  on 
the  meaning  of  the 
requirements. 

•  Review  of  Product  Backlog  (requirements)  with 
Product  Owner  and  team. 

SP  1.2 

Obtain  commitment  to  the 
requirements  from  the  project 
participants. 

•  Release  Planning  and  Sprint  Planning  sessions  that 
seek  team  member  commitment. 

SP  1.3 

Manage  changes  to  the 
requirements  as  they  evolve 
during  the  project. 

•  Add  requirements  changes  to  the  Product  Backlog. 

•  Manage  changes  in  the  next  Sprint  Planning 
meeting. 

SP  1.5 

Identify  inconsistencies  between 
the  project  plans  and  work 
products  and  the  requirements. 

•  Daily  Standup  Meeting  to  identify  issues. 

•  Release  planning  and  Sprint  Planning  sessions  to 
address  inconsistencies. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining. 

•  Release  Burndown  chart  that  tracks  story  points  that 
have  been  completed.  This  shows  how  much  of  the 
product  functionality  is  left  to  complete. 

No  traceability  in  Scrum 

[SP  1.4  Maintain  bidirectional  traceability  among  the  requirements  and  work  products] 
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pp 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Establish  a  top-level  work 
breakdown  structure  (WBS)  to 
estimate  the  scope  of  the  project. 

•  The  standard  tasks  used  in  a  Scrum  process 
combined  with  specific  project  tasks  (Scrum 
Backlog). 

SP  1.2 

Establish  and  maintain  estimates 
of  the  attributes  of  the  work 
products  and  tasks. 

•  Story  Points,  used  to  estimate  the  difficulty  (or 
relative  size)  of  a  Story  (requirement). 

SP  1.3 

Define  the  project  life-cycle 
phases  upon  which  to  scope  the 
planning  effort. 

•  The  Scrum  process. 

SP  1.4 

Estimate  the  project  effort  and 
cost  for  the  work  products  and 
tasks  based  on  estimation 
rationale. 

•  Scrum  Ideal  Time  estimate  (similar  to  billable 
hours  or  Full-time  Equivalents). 

SP  2.1 

Establish  and  maintain  the 
projectO&udget  and  schedule. 

•  Scrum  estimates  (in  Ideal  Time). 

•  Estimates  of  what  work  will  be  in  each  release. 

•  Sprint  Backlog. 

•  Project  Taskboard. 

SP  2.4 

Plan  for  necessary  resources  to 
perform  the  project. 

•  Scrum  estimates  in  Ideal  Time 

•  Release  Plan,  Sprint  Backlog  and  assignments. 

SP  2.6 

Plan  the  involvement  of  identified 
stakeholders. 

•  Scrum  process  roles  (including  team,  Scrum 

Master,  Product  Owner). 

•  [Note:  The  stakeholders  listed  in  Scrum  might  not 
be  the  complete  list  of  stakeholders  for  the  project, 
e.g.,  customers,  other  impacted  teams.] 

SP  3.2 

Reconcile  the  project  plan  to 
reflect  available  and  estimated 

resources. 

•  Sprint  Planning  meeting. 

•  Daily  Scrum  meeting. 
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PMC 

CMMI  Practice 

Scrum  Practice 

SP  1.1 

Monitor  the  actual  values  of 
the  project  planning 
parameters  against  the 
project  plan. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining. 

•  Release  Burndown  chart  that  tracks  completed  story 
points.  This  shows  how  much  of  the  product 
functionality  is  left  to  complete. 

•  Project  Task  Board  used  to  track  stories 
(requirements)  that  are  done,  in  progress,  or  ones 
that  need  verification. 

SP  1.2 

Monitor  commitments  against 
those  identified  in  the  project 
plan. 

•  Discussions  on  team  commitments  at  the: 

-  Daily  Scrum  meeting. 

-  Sprint  Review  meeting. 

•  Sprint  Burndown  chart  that  tracks  effort  remaining. 

•  Release  Burndown  chart  that  tracks  Story  Points  that 
have  been  completed.  This  shows  how  much  of  the 
product  functionality  is  left  to  complete. 

SP  1.6 

Periodically  review  the 
project's  progress, 
performance,  and  issues. 

•  Daily  Scrum  meeting. 

•  Sprint  Review  meeting. 

•  Retrospectives. 

SP  2.3 

Manage  corrective  actions  to 
closure. 

•  Tracking  of  actions  from: 

-  Daily  Scrum  meeting. 

-  Sprint  Review  meeting. 

•  [Note:  This  assumes  that  teams  will  track  (and  not 
lose)  actions.] 

No  risk  assessment  /  tracking  in  Scrum 

[SP  1 .3  Monitor  risks  against  those  identified  in  the  project  plan] 
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Burndown  Charts 


Release  Burndown  Chart 


Implements  PMC  spl.1 

Monitor  the  actual  values  of  the  project  planning  parameters  against  the  project  plan. 


©  Copyright  2004-2010  The  Process  Group.  All  rights  reserved. 


www.processgroup.com 


Version  1.6 


15 


THE 

PRC 

GRC 


Measurement  and  Analysis 


SP  1 .2 

Specif  m  easures  to 
address  the 
measurement 
objectives. 

•  S  p  ri  rh  Birndown  chart  that  tracks  effort 
re  m  a  inn  g. 

•  Release  Bumdcwn  chart  that  tacks  story 
poirts  that  have  been  com  plted.  This  shvos 
hew  much  of  the  poductfunctonaliy  is  leftto 
com  plte. 

•  [Not:  These  two  measures  cold  be  usedto 
track  lie  progress  of  dec  bred  propet 
objectives,  such  asfpntimpQand  f£)n 
budget.O] 

SP  1 .4 

Specif  hew 
measurement  data 
will  beanalyzed  and 
reported. 

•  The  Scrum  prooessdoes  descirbe the  purpose 
and  use  the  Sprint  and  ReleaseBurndown 
charts. 

•  [Nofe:  CM  M  expects  clears  defined  analysis]. 

SP  2.1 

Obtan  speciifed 
measurement  data. 

•  Daty  Scrum  m  eeitig  where  Sprint  and  Relase 
Bundcwn  data  are  collected . 

SP  2.2 

Ana^ze  and  interpret 
measurement  data. 

•  Daty  Scrum  meeihg  where  Sprint  and  Relese 
Bundcwn  date  are  analyzed . 

SP  2.4 

Report  resultsof 
measurement  and 
ana^sisactivites  to 
all  r4evant 
stakeholders. 

•  Daty  Scrum  m  eeitig  where  Sprint  and  Relase 
Bundcwn  charts  are  reiewed. 

•  [Not:  Not  all  interested  stakeholders  will 
necessarily  be  at  theScrum  meeting.] 
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How  About  the  Other 
Components  of  Level  2? 

•  Configuration  Management  (CM): 

-  CM  is  not  specifically  called  out  in  Scrum.  However,  in  an  Agile 
environment  it  is  pretty  easy  to  add  a  layer  of  CM  to  protect  your 
work. 

•  Product  and  Process  Quality  Assurance  (PPQA): 

-  Some  basic  PPQA  activities  are  being  done  naturally  when  the 
Scrum  Master  checks  that  the  Scrum  process  is  being  followed. 

-  Scrum  does  not  specifically  call  out  a  level  of  objective  process  and 
product  check,  nor  does  it  state  that  particular  standards  or 
processes  should  be  defined  and  used. 

•  Supplier  Agreement  Management  (SAM): 

-  Not  included  in  Scrum. 
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Generic  Practices? 

•  Approximately  half  of  the  Level  2  GPs  of  REQM,  PP,  PMC  and  MA 
are  implemented  by  Scrum. 


GP  2.2 

Establish  and  maintain  the  plan 
for  performing  the 
REQM/PP/PMC/MA  process. 

•  The  Scrum  lifecycle  definition  and 
the  milestones  to  perform  Scrum. 

GP  2.3 

Provide  adequate  resources  for 
performing  the 

REQM/PP/PMC/MA  process, 
developing  the  work  products, 
and  providing  the  services  of  the 
process. 

•  The  resources  and  schedule  time 
allocated  to  perform  Scrum  planning, 
monitoring  and  requirements 
activities. 

GP  2.4 

Assign  responsibility  and  authority 
for  performing  the  process, 
developing  the  work  products, 
and  providing  the  services  of  the 
REQM/PP/PMC/MA  process. 

•  The  resource  assignments  allocated 
to  perform  Scrum  planning, 
monitoring  and  requirements 
activities. 

GP  2.6 

Place  designated  work  products 
of  the  REQ  M/PP/PMC/MA 
process  under  appropriate  levels 
of  control. 

•  [Note:  Scrum  does  not  explicitly 
require  CM  to  be  done.  However, 
this  can  be  performed  using  a  digital 
camera,  backed  up  drive,  or  share 
drive  with  versioning  and  controls 
turned  on.] 

GP  2.8 

Monitor  and  control  the 
REQM/PP/PMC/MA  process 
against  the  plan  for  performing 
the  process  and  take  appropriate 
corrective  action. 

•  Scrum  Master  monitoring  that  the 
steps  of  Scr  urn  are  followed. 
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How  About  Level  3? 

•  The  following  Level  3  components  are  not  readily  implemented  by 
Scrum  without  additional  work: 

-  Organizational  Process  Focus 

-  Organizational  Process  Definition 

-  Organizational  Training 

-  Integrated  Project  Management 

-  Risk  Management 

-  Decision  Analysis  and  Resolution 

-  Engineering  PAs  (e.g.,  RD,  TS,  PI,  VER,  VAL) 

-  Generic  Goal  3  (i.e.,  using  an  organization-wide  and  tailored 
process  with  measurements  and  lessons-learned) 
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Scrum  +  -’s 

+  2-4  week  cycles  creates  team  momentum,  and 
early  feedback  on  progress  and  technical 
solutions. 

+  Scrum  process  can  be  learned  and  used  in  less 
than  2  days. 

—  Speed  can  be  mistaken  for  progress: 

-  There  is  no  “Get  good  requirements”  phase,  only 
“Get  a  list  of  1 -liners  and  prioritize.”  (Although  some 
teams  do  more  than  that.) 

-  There  is  no  architecture  /  analysis  phase,  so  you 

could  implement  yourself  into  a  corner.  r* 

-  This  is  fixable  by  making  the  focus  of  each  Sprint^ 
different. 

—  Applying  Scrum  to  large  teams  and  systems  takes 
extra  work. 

-  e.g.,  System  definition,  integration  and  coordination. 
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Summary 

*  Scrum  is  a  good  implementation  for  many  of  the 
practices  in  Level  2. 

*  A  group  can  easily  use  Scrum  and  CMMI  together. 

*  All  the  remaining  practices  in  Levels  2  and  3  can  be 
implemented  while  using  Scrum. 

*  An  organization  at  Level  2  or  3  could  adopt  Scrum  as 
an  additional  lifecycle  choice. 
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Please  send  to  the  authors  if  you  are  interested 
in  reading  the  full  paper  pertained  to  this 
presentation 
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Feasibility  of  Agile  and  CMMI 


•  "The  two  methods  are  certainly  compatible"  Mike 

Konrad,  Senior  Member  of  Technical  Staff  Software  Engineering  Process  Management 
Program 

•  "It  was  an  opportunity  for  us  at  SEI  to  dispel  some 
myths  and  'apologize',  in  a  sense,  to  some  people 
in  the  Agile  community”  Mike  Konrad 


•  Why  Not  Embrace  Both  technical  report  by  SEI 


•  Jeff  Sutherland  Recommends  Combining  Scrum 

with  CMMI  Level  5  Jeff  Sutherland  is  one  of  scrum  inventors 


•  And  Others  . .  .  (www) 
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Feasibility  of  Agile  and  CMMI 
(Cont’d) 


•Our  Own  Various 
Implementations  verify  the 

feasibility! 
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(Cont’d) 

They  are  Synthetically  welded? 


Permission  of  image  copy  is  down  this  slide 
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(Cont’d) 

But,  they  are 
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Why  Agile  CMMI  ? 


•  Easily  understanding  of  model  intent, 

•  Thinking  in  terms  of  valuable 
compliance  instead  of  just  compliance 
or  filling  contract  requirement, 

•  Paying  more  attention  to  people, 

•  Manage  the  fast  deliverables. 
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Why  Agile  CMMI  ?  (Cont’d) 

■j  •  Flow  of  Requirements  is  a  value-driven  to  the 
client 

|  •  Simple  Example:  Conference  Website 

|  °  Elements:  About  us,  Contact  us,  Status  of 

Papers,  Call  for  Papers,  Registration,  and 
Facilities 


Registration 
and  Call  for 
Papers  $$ 


Status  of 
Papers  and 
Facilities 


About  Us 
and  Contact 
Us  $ 


CMMI  10th  Annual  Technology  Conference  and  User  Group 
Association 


-7- 


NDIA-  National  Defense  and  Industrial 


Why  Agile  CMMI  ?  (Cont’d) 


Kuw  ua  do  l:  -aliss  lib  huw  to 
perfomru'conduci  a  process 
through  many  different  and 
unlimited  practices 


Agile  practices,  which  do  rot 
viotaie  Agile  values  antf 
principles 


Agile 

— 

CMMI 

Whfli  lo  do  the  modeJ  of 
CMMl  {Sped  Tic  and  Generic 
Practices)  ask£  us  whaL  Lo 
da  regardless  o1'  the  way  of 
do»ng  it 


N 


CMMl  PA  Goals  (Specific 
and  Genenc  Practices) 


Brief  ReieLidnship  between  AgiSe  atid  CMMI  rnun  implerTientrStinn  perspective 
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Why  Agile  CMMI  ?  (Cont’d) 

•  Heavy-weight  processes  lead  to: 

°  a  late  delivery, 

°  an  inaccurate  assurance  of  quality, 

°  a  misleading  to  the  required  value, 

°  a  more  budget, 

°  a  lot  of  work  around  by  stakeholders  to 
survive! 

•  Remember,  there  are  some  competitors 
over  there  ! 
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Why  Agile  CMMI  ?  (Cont’d) 


1970.  "Managing  the  Development  of  Large 
Software  Systems:  Concepts  and  Techniques".  In: 
Technical  Papers  of  Western  Electronic  Show  and 
Convention  (WesCon)  August  25-28,  1970,  Los 

Angeles,  USA. 


ANALYSIS 


SYSTEM 

REQUIREMENTS 


SOFTWARE 

requirements 


program 

DESIGN 


Dr.  Winston  W.  Royce 
The  Waterfall  Model 


TESTING 


44  I  believe  in  this  concept,  but  the  implementation 

described  above  is  risky  and  invites  failure. 


OPERATIONS 
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Why  Agile  CMMI  ?  (Cont’d) 


Feature  Usage 


Nev 


Source:  Jim  Johnson  of 
the  Standish  Group, 
Keynote  Speech  XP  2002 


Iways 


Often 


Sometimes 


Rarely _ 

0  Always _ Often _ #  Sometimes _ ♦  Rarely _ >  Never 


Dr.  Sidk,y:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 


Feature  Usage 


N I # 


45% 


7°/c 


Source:  Jim  Johnson  of 
the  Standish  Group, 
Keynote  Speech  XP  2002 


Iways 


•  We  need  to  focus  more  of  prioritizing  features  and  delivering  value  quicker  than 
focusing  on  implementing  all  the  features 

•We  need  to  focus  on  the  ROI  of  each  feature  for  your  customer  with  the  help  of 
customer 

•We  need  to  focus  on  the  ROI  of  each  feature  as  a  product  manager 


Sometimes 


Rarely _ 

0  Always _ Often _ #  Sometimes _ #  Rarely _ >  Never 


Dr.  Sidk,y:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 

Challenged  Projects 

m 


USA:  $80-145  billion  per 
year  is  spent  on  failed 
and  cancelled  projects 

UK  :  1 2  out  of  18  Large 
IT  projects  have  failed 


Standish  Report  -  2006 

Dr.  Sidky:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 

However, . . . 

Market  value  of  publicly  traded  shares: 

SNA  (31  December  2009  est.) 
country  comparison  for  the  world 

$34.95  trillion  {31  December  2008) 

$64.56  trillion  <31  December  2007  est.) 


Industries: 


dominated  by  the  onrush  of  technology,  especially  in  computers,  robotics, 


telecommunications,  and  medicines  and  medical  equipment;  most  of  these  advances  take 
place  in  OECD  nations;  only  a  small  portion  of  non-OECD  countries  have  succeeded  in 
rapidly  adjusting  to  these  technological  forces;  the  accelerated  development  of  new 
industrial  (and  agricultural)  technology  is  complicating  already  grim  environmental  problem 


Industrial  production  growth  rate: 

CIA  World  Factbook-  2010 

Dr.  Sidky:  Introduction  to  Agile 
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Why  Agile  CMMI  ?  (Cont’d) 

However, . . . 

Market  value  of  publicly  traded  shares: 

SNA  (31  December  2009  est.) 
country  comparison  for  the  world 


•What  if  we  increased  our  IT  profits  to  both  vendor  and  client  ? 

•What  if  we  eliminated  (or  decreased)  the  challenged  ,  failed  and  canceled  projects? 


dominated  by  the  onrush  of  technology,  especially  in  computers,  robotics, 


telecommunications,  and  medicines  and  medical  equipment;  most  of  these  advances  take 
place  in  OECD  nations;  only  a  small  portion  of  non-OECD  countries  have  succeeded  in 
rapidly  adjusting  to  these  technological  forces;  the  accelerated  development  of  new 
industrial  (and  agricultural)  technology  is  complicating  already  grim  environmental  problem 


Industrial  production  growth  rate: 

CIA  World  Factbook-  2010 

Dr.  Sidky:  Introduction  to  Agile 

dMMI  1 0th  Annual  Technology  Conference  and  User  Group  - 1 5  -  NDIA  -  National  Defense  and  Industrial 
Association 


What  is  “Agile  CMMI  Practice”  or 
ACP? 

•  Agile  Practice  is  not  only  Scurm  or  XP 
practice! 

•  Agile  CMMI  Practice  (ACP)  is  the 
practice  (or  way  of  doing  an  activity) 
that  achieves  the  Specific  Practice 
(SP)  in  CMMI  Model  if  and  only  if  this 
practice  achieves  at  least  one  of  Agile 
Values  and  Principles  without  violating 
any  of  them. 
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In  7  Steps,  how  do  you  achieve 
ACP? 

1.  Make  sure  of  your  senior  management  adoption, 

2.  Discuss  the  policy  and  process  improvement  project  vision 
by  reaching  consensus  with  all  stakeholders  about  the 
vision  and  goals  of  this  project, 

3.  Specify  the  process  areas  in  scope  (i.e.  what?), 

4.  Meet  the  users  (actual  implementers)  of  each  practice  in  the 
selected  process  areas, 

5.  Explore  ,with  the  implementers,  all  possible  valuable  ways 
of  doing  these  practices  which  do  not  violate  any  of  Agile 
Values  and  Principles, 

6.  Consolidate  the  output  from  users  to  reach  the  way  of 
implementing  the  selected  process  areas  and  implicitly  its 
practices  (i.e.  how?) 

7.  Inspect  and  adapt  (i.e.  process  improvement  cycle  is  always 
active) 
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Difference  Between  “Convey” 
and  “Preserve”  Information 


“Convey”  Information 

“Preserve”  Information 

Why? 

Transfer  information  within  your  team  members  for 
alignment  and  having  the  same  understanding  of  this 
information. 

Store  information  for  further  references,  reviews,  reuses  and 
organization  history. 

How? 

It  depends  on  the  case  you  have,  that's  why  Agile  is 
human-centric,  you  choose  the  best  way  for  conveying 
information  in  your  case.  And  do  not  forget  that  one  of 
Agile  principles  is  "the  most  efficient  and  effective 
method  of  conveying  information  to  and  within  a 
development  team  is  face-to-face  conversation", 
however,  if  there  are  rigid  constraints  in  achieving  this 
way,  move  to  the  next  communication  way  which  invites 
interaction  within  team  members.  This  encourages  you  to 
achieve  the  first  step  of  SAMI  (Sidky  Agile  Measurement 
Index),  it  is  a  5-step  value-based  roadmap  to  agility  and 
designed  to  help  guide  organizations  seeking  to  become 
more  agile.  Each  of  SAMI's  5  steps  (Collaborative, 
Evolutionary,  Integrated,  Adaptive  and  Encompassing) 
aims  to  instill  a  new  value  in  teams  and  organizations. 

Also,  it  depends  on  the  case.  Usually,  people  prefer  using  different 
types  of  documentations  project  management  tools,  document 
management  systems  and  we  prefer  using  a  system  that  supports 
your  files'  configurations  such  as  version  history  details. 

When? 

In  every  time  the  team  members  communicate  any 
information  regarding  the  project 

In  all  legal  and  formal  communications.  And  when  you  want  to 
ensure  commitment  and  understanding  of  any  information  or 
responsibilities  with  either  the  customer  or  project  team  members. 
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Assumption 

The  remaining  part  of  this  presentation  mainly  assumes  that: 

•  you  know  and  understand  Agile  scrum  and  XP  practices, 

•  you  understand  REQM  and  RD  Specific  Practices, 

•  a  project/task  management  tool  is  (or  will  be)  used  in  your 
organization, 

•  you  perform  User  Acceptance  Testing  (UAT)  at  the  end  of 
your  release, 

•  if  you  do  not  apply  Test  Automation  or  Test  Driven 
Development  (TDD),  there  is  a  release  stabilization  to 
perform  system  and  regression  testing, 

•  user  story,  beside  its  being  a  valuable  feature  to  the 
customer,  it  can  be  a  non-functional  requirement , 

•  and  all  requirements  are  represented  in  user  stories. 
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(1/3) 

The  purpose  of  Requirements  Development  (RD)  is  to  produce  and  analyze 
customer,  product,  and  product  component  requirements. 


SP# 

CMMI  Specific  Practice 

Agile  CMMI  Practice  (ACP) 

1.1 

Elicit  stakeholder  needs,  expectations, 
constraints,  and  interfaces  for  all  phases  of 
the  product  lifecycle. 

All  customer  meetings  for  gathering  requirements: 
Gathering  of  High  level  stories,  External 

Chartering  (sometimes  called  “Customer/project 
Kickoff”),  Scope  of  Work,  Release  and  Iteration 
stories. 

External  Chartering:  or  customer/project  kickoff  is 
to  make  both  service  provider  and  customer 
aligned  with  the  same  vision,  goals  and  its 
measure  of  success,  project  plan,  stakeholder’s 
roles  and  responsibilities. 

1.2 

Transform  stakeholder  needs, 
expectations,  constraints,  and  interfaces 
into  customer  requirements. 

All  customer  meetings  for  gathering  the 
requirements:  Gathering  of  High  level  stories, 
External  Chartering  (sometimes  called 
“Customer/project  Kickoff”),  Scope  of  Work, 
Release  and  Iteration  (or  Sprint)  stories 

2.1 

Establish  and  maintain  product  and  product 
component  requirements,  which  are  based 
on  the  customer  requirements. 

Scope  of  Work,  Release  and  Iteration  stories 

2.2 

Allocate  the  requirements  for  each  product 
component. 

Release/Iteration  Planning  and  Stories;  by 
grouping  the  related  stories  for  each  product 
component. 
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SP# 

CMMI  Specific  Practice 

Agile  CMMI  Practice  (ACP) 

2.3 

Interfaces  between  functions  (or  between 
objects)  are  identified. 

Project  release  stories. 

3.1 

Establish  and  maintain  operational 
concepts  and  associated  scenarios. 

Project  team  discusses  the  user  stories  and  its 
business  acceptance  tests  (sometimes  called 
“Business  Acceptance  Criteria”)  with  the  business 
analyst  and/or  the  customer  directly  (if  business 
analyst  only,  the  business  analyst  has  to  discuss 
and  confirm  these  user  stories  and  acceptance 
tests). 

3.2 

Establish  and  maintain  a  definition  of 
required  functionality. 

After  reaching  consensus  and  alignment  with  the 
customer  regarding  the  user  stories  (project 
features  and  non-functional  requirements), 
communicate  them  to  the  team,  and  preserve 
them  in  a  maintainable  documentation. 

3.3 

Analyze  requirements  to  ensure  that  they 
are  necessary  and  sufficient. 

Each  user  story  includes  business  acceptance 
tests  which  are  subjected  to  testing  during  the 
iteration,  the  comments  of  your  customer  in 
iteration  demo  if  possible,  and  UAT  feedback. 
These  check-points  of  requirements’  sufficiency, 
significantly,  decrease  the  uncertainty  of  the 
target  value  to  the  customer  (i.e.  reaching  what 
the  customer  needs  more  than  what  the  customer 
“wants”).  From  high  level  view:  the  kickoff 
meetings  that  include  the  main  goals  of  the 
project  support  this  practice. 
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SP# 

CMMI  Specific  Practice 

Agile  CMMI  Practice  (ACP) 

3.4 

Analyze  requirements  to  balance  stakeholder 
needs  and  constraints. 

Each  user  story  includes  business 
acceptance  tests  which  are  subjected  to 
testing  during  the  iteration,  the  comments  of 
your  customer  in  iteration  demo  if  possible, 
and  UAT  feedback.  Meeting  the 
stakeholders  in  kickoffs,  demos,  planning 
and  status  review  meetings  help  you 
balance  the  needs  and  constraints. 

3.5 

Validate  requirements  to  ensure  the  resulting 
product  will  perform  as  intended  in  the  user's 
environment. 

Each  user  story  includes  business 
acceptance  tests  which  are  subjected  to 
testing  during  the  iteration,  the  comments  of 
your  customer  in  iteration  demo  if  possible, 
and  UAT  feedback.  Meeting  the 
stakeholders  in  kickoffs,  demos,  planning 
and  status  review  meetings  help  you 
validate  your  customer  requirements  (i.e. 
make  sure  that  the  requirements  you  are 
developing  are  your  actual  customer  needs, 
and  accordingly,  the  project  meets  your 
customer  expectation.  One  of  the  four  agile 
values  [2]  is  “Customer  Collaboration  over 
Contract  Negotiation”  which  means 
collaboration  in  understanding  the 
requirements  as  well. 

:  CMM 
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SP# 

CMMI  Specific  Practice 

Agile  CMMI  Practice  (ACP) 

1.1 

Develop  an  understanding  with  the  requirements 
providers  on  the  meaning  of  the  requirements. 

All  planning  meetings  (Project,  Release, 
Iteration  and  even  daily  standup  meetings), 
and  we  prefer  that  the  business  analyst  role 
(or  customer  representative)  should  be 
involved  in  such  meetings  at  least  on 
iteration  level;  this  ensures  that  the  team 
members  have  the  same  understanding  of 
requirements  as  the  customer  does. 
Moreover,  iteration  demos  are  key  practice 
for  understanding  customer  requirements. 

1.2 

Obtain  commitment  to  the  requirements  from  the 
project  participants. 

We  encourage  agile  practitioners  to  pay 
more  attention  to  this  practice.  And  as  we 
used  to  say,  do  your  way  of  taking 
commitment  from  both  customer  and 
project  team  members  but  make  sure  that 
this  way  is  effective  when  the  situation  goes 
to  any  kind  of  conflict.  One  of  the  ways  is 
using  formal  emails  or  even  audio/video 
recording  with  your  customer,  and  for 
project  team  members  the  way  should  be 
more  smooth  such  as  project/task 
management  tool,  standup  meetings, 
iteration  plans... etc. 
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IVI 


Specific  Practice 


Agile  CMMI  Practice  (ACP) 


Manage  changes  to  the  requirements  as  they 
evolve  during  the  project. 


In  your  project/task  management  tools, 
differentiate  between  the  first  (original) 
requirements  and  other  further  changes 
even  by  labeling  them.  Try  to  make  the  view 
easy  to  be  tracked  (i.e.  building  a  reportable 
system;  for  example,  in  anytime  after  one 
year  of  production  in  a  project,  you  can 
know  what  the  changes  are,  their  status, 
their  relationships  with  each  other  and  with 
the  first  requirements) 


1.4 


Maintain  bidirectional  traceability  among  the 
requirements  and  work  products. 


Without  this  practice,  most  probably  you  will 
fail  in  managing  your  software  projects 
especially  medium/  large  projects. 

Your  requirements  should  be  in  one  body 
with  all  other  elements  (test  cases, 
architecture,  tasks... etc),  and  make  sure 
that  user  stories,  architecture,  business 
acceptance  tests,  test  cases,  technical 
tasks,  bugs,  and  project  plans  are  all 
traceable  and  connected.  An  easy  way  to 
do  that  is  using  any  agile  project/task 
management  tool  (there  are  a  lot  of  open 
source  and  commercial  products  that  will 
serve  you). 
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SP# 

CMMI  Specific  Practice 

Agile  CMMI  Practice  (ACP) 

1.5 

Identify  inconsistencies  between  the  project  plans 
and  work  products  and  the  requirements. 

In  one  body  of  requirements,  you  can  easily 
identify  the  inconsistencies  (usually,  a 
tracking  tool  is  used) 
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Requirements  Traceability 
Body  in  Agile  (RTBA) 
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figure  2  Requ irefnejii?  irsceabHlty  is  a  human  ixxiy,  his-  land  is  the  planning,  and  the  hal  of 
changes  coma  to  proiati  the  body  from  ilia  deviation  In  hir-;  *-ay  to  ir&aiu^  N&  product  fm 

CMMI  10th  Ann  valuable  to  his  client 

Association 


se  and  Industrial 


Important!  Conclusion 


It  is  not  only  possible  to  embrace  both  agile  and  CMMI,  but  also  it 
is  easier  and  proven  practically.  The  presentation  is  not 
supposed  to  show  the  best  practices  for  your  organization,  the  fore 
mentioned  practices  may  not  work  in  your  organization. 

However,  inspect  and  adapt  to  reach  the  most  appropriate 
practice  that  adds  the  best  possible  value  to  your  organization 
and  your  customer.  Also,  because  agile  mindset  supports  CMMI 
model  intent,  we  found  a  light-weight  implementation  that  we 
cannot  work  without. 
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Thank  You! 

Further  Questions  ? 
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*  Foundations  of  Agile 
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*  Some  ‘Agile,  but’  examples 
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The  Foundations  of  Agile 


Stakeholder 

Value 


Product 

Backlog 


Clear 

Product 

Vision 


User  Stories 

Continuous 

Planning 

Iteration 

Planning 


Co-located 
Teams 


Short  Delivery 
Cycles 


Technical 

Excellence 


Active 

Stakeholder 

Participation 


Visible 

Progress 

Indicators 


Continuous 

Integration 


Retrospectives 


Daily  Standup 
Meeting 
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Potentially 

Shippable 

Software 


Refactoring 

Adherence 
to 

Standards 

Test-Driven 

Development 

(TDD) 

Pair 

Programming  3 


The  State  of  the  Agile  World 


% 


% 


■  Agile 

No  Formal 
Methodology 

■  Iterative 

■  Waterfall 


Based  on  a  2009 
Forrester  /  Dr  Dobbs 
survey,  Agile  adoption  is 
real. 


■  Scrum 

Scrum/XP  Hybrid 

■  XP 

■  Custom  Hybrid 

■  Lean 

Don't  Know  /  Other 


Based  on  the  Version  One 
2009  Survey,  Scrum  and 
Scrum/XP  hybrid  are  the 
most  popular  approaches. 


. .  Agile  and  CMMI 

are  no  longer  at 
war  with  each 
other. 
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A  Closer  Look 


Agile  Practices  That  Are  Most  Effective  (Top  10  out  of  30) 


Continuous  Integration 
Daily  Stand  Up  Meeting 
Developer  TDD 
Iteration  Planning 
Code  Refactoring 
Retrospectives 
Pair  Programming 
Active  Stakeholder  Participation 
Potentially  Shippable  Software 
Burndown  Tracking 


47% 
47% 
44% 


]43% 


339% 
□  36% 
35% 


28% 
126% 


Source:  Ambysoft  Agile  Practices  2009  Sun/ey,  http://www.ambysoft.com/suiveys/practices2009.html 
Copyright  2009  Ambysoft  Inc. 


5% 
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Agile  Practices  That  People  Want  to  Adopt  (Top  10  out  of  30) 


Acceptance  TDD 
Developer  TDD 
J  Potentially  Shippable  Software 
Database  Refactoring 
Pair  Programming 
Executable  Specifications 
Continuous  Integration 
Database  Regression  Testing 
Collective  Code  Ownership 
JIT  Model  Storming 


Source:  Ambysoft  Agile  Practices  2009  Survey,  http://www.ambysoft.com/surveys/practices2009.html 
Copyright  2009  Ambysoft  Inc. 


☆  =  one  of  the  top  1 0  effective  practices 
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18% 


17% 


15% 


27% 


Agile  Practices  Tried  and  Abandoned  (Top  8  out  of  30) 


Pair  Programming 


Burndown  Tracking 


Potentially  Shippable  Software 


Daily  Stand  Up  Meetings 


Executable  Specifications 


Initial  Estimate  and  Schedule 


Active  Stakeholder  Participation 


Retrospectives 


Source:  Ambysoft  Agile  Practices  2009  Survey,  http://www.ambysoft.com/surveys/practices2009.htmi 
Copyright  2009  Ambysoft  Inc. 


24% 
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Agile  Practices  Most  Difficult  to  Learn  (Top  10  out  of  30) 

Developer  TDD 
AceptanceTDD 

128% 


Pair  Programming 
Initial  Estimate  and  Schedule 
Active  Stakeholder  Participation 
Database  Refactoring 
"Potentially  Shippable  Software 
Executable  Specifications 
Release  Planning 
Continuous  Integration 


17% 


26% 


Source:  Ambysoft  Agile  Practices  2009  Suivey,  http://www.ambysoft.com/surveys/practices2009.html 
Copyright  2009  Ambysoft  Inc. 


Therefore 


*  There  must  be  a  lot  of  projects  that  are  ‘Agile,  but’ 
are  not  using  all  of  the  practices  known  to  be 
effective. 

-  So,  how  do  we  assess  the  risk  for  those 
projects  that  claim  to  be  ‘Agile,  but’  are  not 
using  some  of  the  practices  known  to  be 
effective? 
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Premise: 


*  Let’s  use  CMMI  as  a  cross-check  for  risk. 

-  The  lower  CMMI  level,  the  greater  the  risk. 

-  Failing  to  satisfy  goals  implies  more  risk  than 
failing  to  satisfy  particular  practices. 

*  When  goals  will  not  be  met,  alternative  practices 
will  be  needed. 

-  Hybrid  approaches  may  be  more  appropriate 
than  ‘Agile,  but’. 
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We’re  doing  ‘Scrum,  but’  we’re 
not  doing  retrospectives. 


Retrospectives 


*  What  does  CMMI  say? 

*  CAR  SG  1  Determine  causes  of 
defects. 

•  CAR  SG  2  Address  causes  of 
defects. 

4 

•  Without  retrospectives,  problems 
other  than  those  identified  via 
discrepancy  reports  may  not  be 
identified  or  addressed 
systematically. 
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ii 


We’re  ‘Agile,  but’  we  don’t  have 
active  stakeholder  participation. 


Stakeholder  participation  is 
needed  for  release 
planning,  iteration  planning 
and  demonstrations. 


What  does  CMMI  say? 

>  REQM  SG  1  Manage  requirements. 

>  PP  SG  3  Obtain  commitment  to  the  plan. 

>  IPM  SG  2  Coordinate  and  collaborate 
with  relevant  stakeholders. 

*  RD  SG  1  Develop  customer 
requirements. 

*  RD  SG  2  Develop  product  requirements. 

>  RD  SG  2  Analyze  and  validate 
requirements. 


Stakeholders  also  need  to 
be  available  to  answer 
questions  and  clarify 
requirements. 


We  need  to  ask  what  the  team  is  doing 
for  requirements  management  and 
analysis.  12 
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We’re  doing  ‘XP,  but’  we  arerft  ~ ^7 
doing  pair  programming. 


Pair  programming  gives 
2  sets  of  eyes  on  the 
code  at  all  times. 


•  What  does  CMMI  say? 

•  VER  SG  2  Perform  Peer  Reviews 


Agile  teams  that  do  not  do  pair 
programming  need  some  other  form  of 
peer  review. 

•  Reviewing  every  week  at  a  specific  time. 

•  Reviewing  at  the  end  of  the  iteration. 

•  Using  static  analysis  tools  and  test 
coverage  tools  to  supplement  human 
reviews. 
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We’re  ‘Agile,  but’  we  aren’t  doing 
continuous  integration. 


4iinHnL£Kr_LtfD'ip- 


O1 tee- windows  passed  (44  minutes  and}  (_J 

Du>ld  fiirte:  27  Nov  200?  W:Sl  GMT  +0S  DO  Durdthpil:  7  rwiutei  4d  seconds 

UuiliE:  build.  U 


ILrtll9C.li  Mu-JkI  iLJtiOii*.  Build;  Ldm  Tiisli  Errutc  aiid  Wad-nidQii 

Modifications 

*  bL’^.ll  ri  nrnlchri^  [Chris  A.  Cno  Li"  fix  fid  issue:  with  -■jcuc-r  inactive-  status. 

Jreu .  7847]  _  .  branches/  ccertmisecontrol/repof hnci/deshtjcard  jsumt.1  ts f']son_to_c55_test.htniJ 

ll  t11# .  3847]  j,  /brand  iu/ccc/ jrnisec  «■  Lrol-'i epe-1  ting/dashba ord/wob jcp-'j av jscTpts/j3an_ta_css.p 


l.fftopst  Mllds 
&  7  mihutas  eyrJ  bUikl  .8 
O  44  innutes  ago  bu^J  8 
&  BbOut  17  I  tourf  jyu  buld.7 
D  jbuut  17  hws  a  go 
O  about  18  hWS  ago  Li-ld  6 
O  jbout  19  liL-'ji  '=  ago  buld  S 
C#  about  19  htowS-  ago  btdd.4 
i'j  1  d dy  ago  buJi.l  j 
ij  1  day  t*lu  bulJ-2 
Cz  a  days  aijj  buiU.i 


What  does  CMMI  say? 

•  PI  SG  1  Prepare  for  product  integration. 

•  PI  SG  2  Ensure  interface  compatibility. 

•  PI  SG3  Assemble  product  components  and 
deliver  the  product. 
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Recommendations 


X 


*  When  you  hear  we’re  ‘Agile,  but’,  check  what  the 
CMMI  has  to  say 

-  Is  a  goal  being  missed? 

-  Will  alternative  practices  work? 

-  Is  it  an  OK  modification? 

*  E.g.  virtually  co-located  teams  are  OK  when 
supported  by  a  communications 
infrastructure 
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Purpose 


The  purpose  of  this  presentation  is  to  discuss  the  changes  in  CMMI  VI  .3 
in  the  context  of  how  you  interpret  and  apply  those  changes. 

You  may  ask,  “Now  that  the  CMMI  Product  Suite  was  released,  how  do  I 
use  it?”  By  the  end  of  this  presentation,  I  hope  that  you  have  some  tips 
and  hints  that  you  can  apply  on  your  CMMI  VI  .3  journey. 

Caveat:  There  are  many  different  ways  to  interpret  and  apply  this  material. 

One  of  the  true  advantages  of  using  CMMI  is  the  ability  to  use  this  material 
in  many  different  contexts  and  situations.  Therefore,  only  use  this 
information  if  you  agree  with  it. 

•  If  it  doesn’t  fit  your  organization  understand  why. 

•  Understand  what  you  do  differently. 

•  Adjust  this  guidance  to  fit  your  needs. 
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Topics 


CMMI  Model(s)  Changes 
CMMI-DEV  Addison-Wesley  Book 
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CMMI  VI  .3  Model(s)  Changes 
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Three  Constellations  and  Three  Models: 
Which  Do  I  Choose? 


Because  of  harmonization,  the  models  are  more  similar  in  VI  .3. 


Core  PAs  appear  in  all 
CMMI  models;  however... 

•  These  PAs  are  not 
identical  across  all 
models. 

•  Informative  material  can 
be  different  so  that  users 
interpret  goals  and 
practices  for  their  area  of 
interest. 

•  Sometimes  practices  can 
be  different  in  one  model 
from  another  (i.e.,  PP, 
PMC). 


CMMI-DEV 


CMMI-SVC 


1 6  Core  Process 
Areas 


CMMI-ACQ 


Pick  the  model  that  most  closely  aligns  with  your  business  needs. 
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Picking  the  Right  Model  for  Your  Organization., 


1 

If  you  are  just  starting  process  improvement  (i.e., 
working  for  two  years  or  less), 

•  pick  one  of  the  models  and  use  it, 

•  focus  on  the  core  process  areas,  and 

CMMI*  for  DovNopmont  Vers  ton  1  3 

CMMMJtV  VI  3 

•  remember  that  some  of  the  core  process  areas  are 

slightly  different  (e.g.,  PP  and  PMC  at  the  specific  practice  level, 
names  of  the  PAs). 

If  you  are  a  seasoned  CMMI  user, 

•  use  all  3  models,  especially  if  you  are  improving  many  different 
parts  of  your  organization,  and 

•  look  at  the  “unique”  PAs. 

If  you  have  been  using  only  CMMI-DEV,  you  are  missing  out  on  useful 
practices. 
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Picking  the  Right  Model  for  Your  Organization 


The  VI. 2  models  were  improved  and  released 
incrementally. 

CMMI-DEV  was  released  in  2006. 

CMMI-ACQ  was  released  in  2007. 

Two  years  later,  CMMI-SVC  was  released. 

CMMI-DEV  is  where  the  most  change  occurred  from  VI  .2  to  VI  .3. 

In  many  instances,  changes  to  CMMI-DEV  for  VI  .3  were  already 
incorporated  into  the  other  two  models  as  part  of  their  development  efforts 
for  VI  .2. 

As  you  would  expect,  CMMI-SVC  has  the  least  change. 


CMMI*  for  nwftJaprnen!  1  3 

CMMU-OEV.  Vi  3 
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Understanding  the  Changes 


If  you  are  a  seasoned  CMMI  user,  you  will  find  the  comparison  files  useful. 
They  are  available  at  http://www.sei.cmu.edu/cmmi/tools/cmmiv1- 
3/index.cfm 

All  model  changes  are  included  in  the  comparison  files: 

•  Movement  of  material 

•  Editorial  changes 

•  Harmonization  changes 

Comparison  files  do  not  help  you  find  the  major  changes,  however.  The 
CMMI  VI. 3  Model  Upgrade  Training  is  a  better  source  for  understanding 
high-level  changes. 

Also,  many  of  the  presentations  at  this  conference  address  changes  to 
VI  .3  and  are  good  sources  of  information. 
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Architecture  Changes 


Most  components  did  not  change. 


One  primary  change  was  that 
“typical”  was  replaced  by  “example” 
when  presenting  work  products. 

The  result  should  be  that  this 
component  is  easier  to  interpret  and 
more  usable. 

Your  organization  is  more  likely  to  be 
successful  in  developing  the  work 
products  that  are  right  for  your 
organization  instead  of  those 
presented  in  CMMI. 


Process  Area 


Example  Work 
Products 


Example 

Supplier 

Deliverables 


Subpractices 


(GP) 


Subpractices 


Generic  Practice 
Elaborations 
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Changes  to  PA  Categories 


Service  Establishment 
and  Delivery 

Engineering 

Acquisition 

Engineering 

i  ' 

*  REQM  AM,  SSAD 

Process 

Management 

Project/Work* 

Management 

*CMMI-SVC  only 

A- 

Support 

These  changes  should  provide  a  more  natural  fit  to  align  with  PA 
content. 

If  you  have  tools  or  documents  that  use  PA  categories,  slight 
adjustments  will  be  needed. 
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Changes  to 


Generic  Goals  and  Practices., 


In  CMMI-DEV,  the  location  of  generic  goals  (GGs)  and  generic  practices 
(GPs)  was  changed. 

•  Don’t  let  this  change  minimize  the  importance  of  the  GGs  and  GPs. 

•  Use  this  change  as  an  enabler  to  stress  the  importance  of 
establishing  the  “process  improvement”  infrastructure  in  your 
organization. 

•  If  you  have  different  teams  interpreting  and  applying  PAs,  don’t  let 
them  forget  to  look  at  the  GGs  and  GPs,  especially  the  elaborations. 

—  Remember  that  elaborations  only  exist  in  CMMI-DEV  and  CMMI-SVC. 
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Changes  to 


Generic  Goals  and  Practices2 


Generic  Goals 

Generic  Practices 

GGl:  Achieve 

GP  1.1: 

Perform  Specific  Practices 

Specific  Goals 

GG2:  Institutionalize  a 

GP  2.1 

Establish  an  Organizational  Policy 

Managed  Process 

GP  2.2 

Plan  the  Process 

GP  2.3 

Provide  Resources 

GP  2.4 

Assign  Responsibility 

GP  2.5 

Train  People 

GP  2.6 

Control  Work  Products 

GP  2.7 

Identify  and  Involve  Relevant  Stakeholders 

GP  2.8 

Monitor  and  Control  the  Process 

GP  2.9 

Objectively  Evaluate  Adherence 

GP  2.10:  Review  Status  with  Higher  Level  Management 

GG3:  Institutionalize  a 

GP  3.1: 

Establish  a  Defined  Process 

Defined  Process 

GP  3.2: 

Collect  Improvement  Information 

Adapted  from 
Cepeda  Systems  & 
Software  Analysis,  Inc. 


These  changes  were  always  implied  but  now  they  are  explicitly 
stated. 
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Changes  to 


Generic  Goals  and  Practices3 


The  biggest  change  -  level  4  and  5  generic  goals  and  practices  were 
eliminated  to  appropriately  focus  high  maturity  on  the  achievement  of 
business  objectives. 

Thus,  the  concept  of  capability  levels  4  and  5  was  removed  also. 

GG  4  Institutionalize  a  Quantitatively  Managed  Process 
GP  4.1  Establish  Quantitative  Objectives  for  the  Process 
GP  4.2  Stabilize  Subprocess  Performance 
GG  5  Institutional. 'e  an  Optimizing  Process 
GP  5.1  Ensure  Contir'ious  Proces^rnprovement 
GP  5.2  Correct  Root  Causes  of  Problems 
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What  Happened  to  IPPD  and  Amplifications? 


Fewer  than  5%  of  recent  appraisals  have  included  IPPD.  However,  in 
today’s  environment  most  people  work  in  teams. 

Teaming  has  been  embedded  consistently  in  all  3  models. 

•  This  change  allows  you  to  address  these  concepts  as  intended. 

•  You  don’t  have  to  worry  about  whether  to  include  teaming  practices 
or  not. 

In  CMMI-DEV  there  were  about  20  amplifications  for  SE,  SW,  and  HW  and 
none  in  CMMI-ACQ  or  CMMI-SVC.  Therefore,  amplifications  were 
removed.  Useful  amplifications  were  converted  to  examples  and  notes. 

•  The  concept  of  amplification  was  often  confusing. 

•  This  change  should  allow  CMMI  users  to  look  at  this  information  like 
other  informative  material. 
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Terminology  Was  Improved.] 


CMMI  terminology  evolved  and  wasn’t  always  consistent  with  the  release  of 
new  constellations. 

The  model  component  descriptions  were  revised.  If  defined  in  the  glossary 
the  following  occurred: 

•  The  redundant  text  in  the  description  was  minimized. 

•  Inconsistent  text  was  eliminated. 

•  A  reference  to  the  glossary  was  added. 

Some  glossary  terms  were  not  consistent  with  how  the  same  terms  were 
described  in  other  parts  of  the  model  and  they  did  not  accurately  reflect  their 
relationships  with  other  terms. 

The  glossary  had  a  “good  scrub.” 

Therefore,  when  you  don’t  understand  a  term,  first  refer  to  the  glossary. 
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Terminology  Was  lmproved2 


Certain  words  and  phrases  that  appear  in  goal  and  practice  statements 
unnecessarily  complicate  their  interpretation. 

Words  and  phrases  that  were  considered  problematic  include:  appropriate,  best, 
designated,  effectively,  essential,  most  important,  necessary,  realistic, 
reasonable,  and  relevant. 

When  implementing  these  practices,  it  should  be  a  given  that  you 
implement  what  is  “appropriate”  for  the  project  and  organization. 

This  change  should  minimize  some  of  the  interpretation  wars  that  occurred 
during  appraisals. 
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Informative  Material 


Informative  material  was  improved,  including  revision  of  the  Engineering 
practices  to  reflect  industry  best  practice  and  additional  guidance  for 
organizations  that  use  Agile  methods. 

References  were  revised  so  that  users  can  easily  find  the  information  that 
the  reference  points  to  by  searching  for  a  goal  or  practice  title  or  purpose 
statement  in  the  destination  PA 

A  standard  sequence  of  references  was  introduced  when  there  are 
multiple  adjacent  references: 

•  Constellation-unique  PAs  appear  first. 

•  Within  each  PA  reference  grouping,  the  references  are  listed 
alphabetically  by  the  destination  PA. 
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Front  Matter  Changes 


The  history  of  CMMI  was  added  to  accompany  Figure  1 .2. 

With  a  few  exceptions,  the  mention  of  “source  models”  was  removed. 

Biases  favoring  maturity  levels  or  capability  levels  were  removed. 

Clarifications  were  added  that  CMMI  models  are  not  processes  or  process 
descriptions. 

Information  was  added  about  selecting  the  right  CMMI  model. 

Clarifications  were  added  that  Chapter  2,  Process  Area  Components, 
contains  descriptions,  not  definitions. 

In  DEV  only,  it  is  now  mentioned  that  recursion  among  PAs  can  also  apply 
to  Project  Management  PAs.  (In  VI  .2,  some  inferred  the  idea  of  recursion 
might  apply  to  Engineering  PAs  only.) 

Changes  were  made  to  explicitly  state  what  was  always  implied. 


—  Tips  and  Hints  for  CMMI  VI. 3 

Software  Engineering  Institute  CarnegieMellon  10th  Annual  CMMI  Technology  Conference 


©2010  Carnegie  Mellon  University 


PA  Changes  at  Maturity  Levels  2  and  3 


All  PAs  have  changed. 

Five  PAs  were  changed  at  the  goal  level: 

•  IPM,  IRP,  OPD,  OT,  and  RD 

Fourteen  PAs  were  changed  at  the  specific  practice  level: 

•  IPM,  OPD,  PI,  PPQA,  REQM,  RSKM,  SAM, 

•  SSAD 

•  PI,  RD 

•  IRP,  SCON,  SSD,  SST,  STSM 

Be  sure  to  plan  adjustments  to  your  process  improvement  efforts.  And 
don’t  forget  to  mine  the  changes  to  the  informative  material  to  determine 
how  those  changes  may  benefit  your  organization. 
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High  Maturity  Changes-, 


High  maturity  concepts  and  terminology  were  confusing  in  VI  .2. 

High  maturity  requirements  were  implied,  not  explicitly  stated  and  were 
sometimes  contained  in  documents  other  than  the  model. 

Explanations  were  not  central  or  consistent. 

There  were  inaccurate  perceptions  about  high  maturity. 

High  maturity  concepts  were  present  in  ALL  constellations,  but  in  much  of 
the  material  examples  focused  on  development. 

If  you  are  a  high  maturity  organization  or  are  pursuing  high  maturity,  move 
to  VI  .3  as  quickly  as  possible! 
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High  Maturity  Changes2 


High  maturity  process  areas  are  significantly  improved  to  reflect  industry 
best  practices,  including  a  new  specific  goal  and  several  new  specific 
practices  in  the  process  area  that  was  renamed  from  Organizational 
Innovation  and  Deployment  (OID)  to  Organizational  Performance 
Management  (OPM). 

Organizational  Process  Performance  was  reordered. 

Quantitative  Project  Management  was  reordered  and  modified. 

The  scope  of  Causal  Analysis  and  Resolution  was  changed. 

•  “Defects  and  other  problems”  was  replaced  by  “outcomes,”  which 
can  be  successes  as  well  as  deficiencies. 

•  The  emphasis  on  “proactive”  defect  prevention  was  increased. 
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The  Book  Changes 


The  design  of  the  Addison-Wesley  book  stayed  basically  the  same  as  the 
second  edition  (Versionl.2). 

•  Some  perspectives  stayed  the  same,  some  were  updated,  and  new 
ones  were  added. 

•  The  case  study  was  replaced  with  three 
shorter  case  studies. 

•  Tips  and  hints  were  only  modified  where 
needed.  The  Engineering  and  high  maturity 
PAs  had  the  most  change. 

The  book  is  scheduled  for  release  at  SEPG  North 
America,  March  21-24,  201 1 . 
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Summary 


The  VI  .3  changes  are  improvements  to  the  model. 

The  model  should  be  easier  to  implement  and  interpret  if  you  have  been 
following  sound  process  improvement  principles. 

These  changes  should  not  have  a  noticeable  effect  on  what  you  have 
been  doing. 

They  should  be  more  of  a  natural  fit. 

Does  it  surprise  you  that  most  of  the  change  occurred  in  the  informative 
material? 


—  Tips  and  Hints  for  CMMI VI. 3 

Software  Engineering  Institute  CarnegieMellon  10th  Annual  CMMI  Technology  Conference 


©2010  Carnegie  Mellon  University 


Contact  Information 


Mary  Beth  Chrissis 
SEPM 

Telephone:  412-268-5757 
Email:  mb@sei.cmu.edu 


Software  Engineering  Institute 


U.S.  Mail: 

Software  Engineering  Institute 
Carnegie  Mellon  University 
4500  Fifth  Avenue 
Pittsburgh,  PA  15213-3890 

World  Wide  Web: 

http://www.sei.cmu.edu/productlines 

SEI  Fax:  412-268-5758 
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CMMI-DEV  vl.3  Good?  -  Bad?  -  Ugly? 


Jackie  Lawrence  Certified  High  Maturity  SCAMPI  Lead  Appraiser 

Nathaniel  Becker  US  Army  Armament  SEC  Process  Lead 

Jeanine  Courtney-Clark  Certified  High  Maturity  SCAMPI  Lead  Appraiser 


-  US  ARMY 

0  RDECOM >  Agenda 


Context 

*  Appraisal  Preparation  and  Planning 

*  Appraisal  On-site 

*  Appraisal  Reporting 
Impact  on  Appraisal  Results 

CMMI®-  DEV  vl.3  Practice  Characterizations 
Model  Specific  Recommendations 


•®CMM,  Capability  Maturity  Model,  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by  Carnegie  Mellon  University. 
•sm  CMM  Integration,  SCAMPI,  and  IDEAL  are  service  marks  of  Carnegie  Mellon  University. 
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0  rdecom^  Organizational  Context 


*  Long  (>10  year)  organizational  process  improvement 
legacy. 

*  Appraisal  Sponsor  requested  the  appraisal  team 
pilot  CMMI®-  DEV  vl.3  high  maturity  practices. 

*  Appraisal  objective  -  determine  the  potential  impact 
to  the  organization’s  current  practices. 

*  Repeat  CMMI®  ML  5  appraisal  for  the  organization. 
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*  Appraisal  team  training  supplemented  by: 

-  Most  of  appraisal  team  attended  the  SEPG  2010  Conference 
special  session:  CMMI®-  DEV  vl.3  for  High  Maturity  Lead 
Appraisers,  as  invited  guests. 

-  QPM  and  OPM  webinar  conducted  by  two  members  of  the  SEI 
staff  and  other  CMMI®-  DEV  vl.3  development  team  members 
during  the  appraisal  readiness  review. 

-  Each  team  member  was  required  to  review  the  CMMI®-  DEV  vl.3 
GLOP  provided  to  the  team  by  the  SEI  CMMI®  Program  Manager. 

*  Other  Planning 

-  No  additional  interviews. 

-  No  additional  data  collected. 

-  No  schedule  impact. 
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Jackie  Lawrence  (Team  Lead) 

Nathan  Becker 

Billy  Thomas 

Helen  Huie 

Steve  Lubash 

Timothy  Grealy 

Jeanine  Courtney-Clark 
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CMMI®-  DEV  vl.3  materials  used 


•  GLOP  -  March  22,  2010  (CAR  and  OPP) 

•  QPM  -  April  28,  2010  (email  date) 

•  OID/OPM-May  12,  2010 
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*  Use  the  existing  data  provided  by  the 
organization  for  the  CMMI-DEV  vl.2  appraisal. 

*  Focus  on  ML4  (OPP  and  QPM)  and  ML  5  (OPM 
and  CAR)  process  areas. 

*  Identify  areas  that  may  impact  either 
implementation  or  an  appraisal. 

*  Carefully  review  the  intent  of  the  new  model  text 
and  classify  each  change  as  either  better 
explanations,  new  expectations  (practice 
changes)  or  new  recommendations  (informative 
text). 
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Team  Feedback/Experience: 

“Maintain  Business  Objectives”  (OPM  SP  1.1)  seemed  out  of  sequence.  We 
felt  a  need  to  have  a  practice  in  OPP  to  maintain  objectives  already 
established  (also  see  recommendation  for  OPF). 

The  explanations  in  the  informative  material  are  more  helpful.  Specifically 
the  new  language  in  sub  practices  1-3: 

-  criteria  for  selecting  sub  practices, 

-  traceability  between  sub  processes  and  objectives 

-  using  causal  and  sensitivity  analysis  for  sub  process  selection 

-  determining  a  range  of  expected  performance  results. 

Team  Recommendation: 

Move  SP  1.1  Maintain  Business  Objectives  from  OPM  SP  1.1  to  OPP 
SP  1.1  and  renumber  practices  in  OPP  SG  1. 
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m  RDECOMj  OPP  vl.3  vs  vl.2  Change 
v  Highlights 


vl.3  -  New  SP  1.1  Establish  and  maintain  the  organization’s 
quantitative  objectives  for  quality  and  process  performance,  which 
are  traceable  to  business  objectives. 

vl.3  -  SP  1.2  Select  processes  or  subprocesses  in  the  organization’s 
set  of  standard  processes  to  be  included  in  the  organization’s 
process  performance  analyses  and  maintain  traceability  to  business 

objectives. 

vl.3  SP  1.3  Establish  and  maintain  definitions  of  measures  to  be 
included  in  the  organization’s  process  performance  analyses. 

vl  .2  SP  1 .3  Establish  and  maintain  quantitative  objectives  for  quality  and 
process  performance  for  the  organization. 


vl.3  SP  1.4  Analyze  the  performance  of  the  selected  processes,  and 
establish  and  maintain  the  process  performance  baselines. 


.2  SP  1.4  Establish  and  maintain  the  organization's  process-performance 
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Team  Feedback/Experience: 

There  appears  to  be  significant  overlap  between  QPM  SP  2.1  and  QPM  SP 
2.2  (e.g.,  a  clear  distinction  is  not  made  between  manage  versus  monitor  in 
the  subpractices).  [QPM  SP  2.1,  QPM  SP  2.2] 

We  would  prefer  to  see  some  other  types  of  examples.  As  written  this  the 
model  may  lead  organizations  to  focus  on  PR  too  much.  [QPM  SP  1.2] 

We  suggest  there  should  be  fewer  subpractices  on  measures  and  more  on 
selecting  analytic  techniques.  [QPM  SP  1.4] 

QPM  SP  2.2  subpractice  6  and  QPM  SP  2.3  subpractice  2  are  virtually  the 
same.  [QPM  SP  2.2,  QPM  SP  2.3] 

Team  Recommendation: 


formance  of  the  selected  sub  processes. 
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ro  rdecom  >  QPM  -  vl  .3  vs  vl  .2  Change 
v  Highlights 


vl.3  SP  1.2  Using  statistical  and  other  quantitative  techniques, 
compose  a  defined  process  that  enables  the  project  to  achieve  its 
quality  and  process  performance  objectives. 

vl  .2  SP  1 .2  Select  the  subprocesses  that  compose  the  project’s  defined  process 
based  on  historical  stability  and  capability  data. 

vl.3  SP  1.3  Select  subprocesses  and  attributes  critical  to  evaluating 
performance  and  that  help  to  achieve  the  project’s  quality  and 
process  performance  objectives. 

vl  .2  SP  1 .3  Select  the  subprocesses  of  the  project's  defined  process  that  will  be 
statistically  managed. 

V  1.3  SP  1.4  Select  measures  and  analytic  techniques  to  be  used  in 
quantitative  management. 

vl  .2  SP  1 .4  Monitor  the  project  to  determine  whether  the  project’s  objectives  for 
quality  and  process  performance  will  be  satisfied,  and  identify  corrective  action  as 
appropriate. 
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SP  2.1  Monitor  the  performance  of  selected  subprocesses  using 
statistical  and  other  quantitative  techniques. 

vl  .2  SP  2.1  Select  the  measures  and  analytic  techniques  to  be  used  in  statistically 
managing  the  selected  subprocesses. 

SP  2.2  Manage  the  project  using  statistical  and  other  quantitative  techniques 
to  determine  whether  or  not  the  project’s  objectives  for  quality  and  process 
performance  are  being  satisfied. 

vl.2  SP  2.2  Establish  and  maintain  an  understanding  of  the  variation  of  the  selected 
subprocesses  using  the  selected  measures  and  analytic  techniques. 

SP  2.3  Perform  root  cause  analysis  of  selected  issues  to  address  deficiencies 
in  achieving  the  project’s  quality  and  process  performance  objectives. 

vl  .2  SP  2.3  Monitor  the  performance  of  the  selected  subprocesses  to  determine 
their  capability  to  satisfy  their  quality  and  process  performance  objectives,  and 
identify  corrective  action  as  necessary. 


Integrated  System  Diagnostics 


Malcolm  Baldrige 

National 
Quality 
Award 


2007  Award 
Recipient 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


US  ARMY 

0  rdecom p  Causal  Analysis  and  Resolution 


Team  Feedback/Experience: 

The  words  "including  clear  definition  of  the  improvement  needed  or 
expected"  -  subpractice  3  -  seems  to  anticipate  the  root  causes  before  the 
root  causes  are  determined.  [CAR  SP  1.1] 


Including  positive  outcomes  is  an  improvement  to  the  model.  [CAR  SP  1.2] 


The  new  wording  poses  a  risk  that  organizations  will  be  required  to  show 
evidence  of  analysis  of  both  problem  and  success  outcomes  in  an 
appraisal. 


Implement  the  action  plans  -  sub  practice  4  -  ties  to  the  practice  text  of 
"Implement".  This  is  an  improvement  in  the  wording.  [CAR  SP  2.1] 

Team  Recommendation: 

•  No  recommended  changes  were  given  by  the  team. 
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m  rdecom>  CAR  -  vl  .3  vs  vl  .2  Change 
v  Highlights 


vl.3  SP  1.1  Select  outcomes  for  analysis. 

vl  .2  SP1 .1  Select  the  defects  and  other  problems  for  analysis. 


vl.3  SP  1.2  Perform  causal  analysis  of  selected  outcomes  and 
propose  actions  to  address  them. 

vl  .2  SP  1 .2  Perform  causal  analysis  of  selected  defects  and  other  problems  and 
propose  actions  to  address  them. 

vl.3  SP  2.3  Record  causal  analysis  and  resolution  data  for  use  across 
projects  and  the  organization. 

vl  .2  SP  2.3  Record  causal  analysis  and  resolution  data  for  use  across  the  project 
and  organization. 


Malcolm  Baldrige 

National 
Quality 
Award 


2007  Award 
Recipient 


Integrated  System  Diagnostics 


TECHNOLOGY  DRIVEN.  WARFIGHTER  FOCUSED. 


Team  Feedback/Experience: 

Suggest  moving  OPM  SP  1.1  to  make  it  a  new  OPP  SP  1.1.  Need  to  have  a 
practice  in  OPP  to  revise  the  business  objectives  identified  in  OPF  and 
expanded  in  OPM.  [OPM  SP  1.1] 

Incremental  improvements  are  handled  by  the  pilot  organization  in  OPF, 
and  not  explicitly  called  out  as  relating  to  the  QPPOs  and  business 
objectives  like  an  innovation  would  be.  [OPM  SP  2.1] 

Adding  incremental  and  innovative  improvements  in  two  separate 
practices  may  cause  confusion  for  implementers  and  appraisers. 
Recommend  deleting  this  practice.  With  the  exception  of  tying  them  to 
business  needs,  incremental  changes  are  analyzed  and  implemented  in 
OPF  (see  current  examples  given  in  OPM  SP  2.1  text).  [OPM  SP  2.1] 
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Team  Feedback/Experience: 

Add  Business  Objectives  to  this  practice  in  same  way  that  QPPOs  are  currently 
included.  [OPM  SP  2.5] 

Team  Recommendation: 

•  Move  SP  1.2  Analyze  Process-Performance  Data  to  SG  2  SP  2.1. 

•  Remove  SP  2.2  Elicit  and  Analyze  Incremental  Improvements. 

•  Remove  SP  2.3  Implement  Improvements.  [Note:  The  GLOP  said 
“implement”  and  SP  2.5  said  “deploy”] 
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SG  1  Manage  the  organization’s  business  performance  using 
statistical  and  other  quantitative  techniques  to  understand 
process-performance  shortfalls  and  identify  areas  for  process 
improvement. 

-  SP1.1  Maintain  business  objectives  based  on  an  understanding 
of  business  strategies  and  actual  performance  results. 

-  SP  1.2  Analyze  process-performance  data  to  determine  the 
organization’s  ability  to  meet  identified  business  objectives. 

-  SP1.3  Identify  potential  areas  for  improvement  that  could 
contribute  to  meeting  business  objectives. 

Team  Recommendation: 

•  Remove  SG  1  and  associated  specific  practices  and  renumber  SG  2 
and  SG  3. 
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SG  2  Improvements  are  proactively  identified,  evaluated  using 
statistical  and  other  quantitative  techniques,  and  selected  for 
deployment  based  on  their  contribution  to  meeting  quality  and 
process-performance  objectives. 

-  SP  2.1  Elicit  and  categorize  suggested  improvements. 

-  SP  2.2  Analyze  suggested  improvements  for  their  possible  impact 
on  achieving  the  organization’s  quality  and  process-performance 
objectives. 

-  SP  2.3  Validate  selected  improvements. 

-  SP  2.4  Select  and  implement  improvements  for  deployment 
throughout  the  organization  based  on  an  evaluation  of  costs, 
benefits  and  other  factors. 

Team  Recommendation: 

•  Rename  proposed  current  SG  2  to  Manage  Business  Performance 
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SG  3  Measurable  improvements  to  the  organization’s 
processes  and  technologies  are  deployed  and  evaluated  using 
statistical  and  other  quantitative  techniques. 

-  SP  3.1  Establish  and  maintain  plans  for  deploying  selected 
improvements. 

-  SP  3.2  Manage  the  deployment  of  selected  improvements. 

-  SP  3.3  Evaluate  the  effects  of  deployed  improvements  on  quality 
and  process  performance  using  statistical  and  other  quantitative 
techniques. 
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Organizational 

OPF 


Process  Focus  - 


Feedback: 


If  OPF  SP  1.1  were  changed  to  read  “Establish  and  maintain  business 
objectives"  this  would  better  support  "Maintain  business  objectives  “ 
in  OPP  and  "Analyze  the  process-performance  data”  in  QPM. 


Team  Experience: 

•  The  organization  of  the  SPs  between  OPF,  OPP  and  OPM  was 
confusing  for  the  team. 

•  It  was  the  team’s  view  that  because  an  SP  to  establish  the  business 
objectives  does  not  appear  in  ML3  that  this  might  cause  a 
disconnect  in  implementation  (e.g.,  the  model  says  maintain 
objectives  in  OPP  and  use  the  business  objectives  OPM  but  did  not 
clearly  establish  the  objectives  beforehand). 


Team  Recommendation 

Replace  SP1.1  with  “Establish  and  Maintain  Business  Objectives” 
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0  rdecom p  Appraisal  Reporting 


•  The  team  created  a  detailed  feedback  report  including  redlines 
for  each  process  area  of  the  CMMI®-  DEV  vl  .3  materials  that  we 
were  provided. 

•  Collaborating  to  build  a  report  that  included  information  from 
both  an  appraisal  team  and  the  organization  allowed  us  to 
provide  detailed  feedback  while  taking  both  an  appraisal  and 
implementation  perspective. 

•  Mapping  of  CMMI®-  DEV  vl  .2  to  CMMI®-  DEV  vl  .3  was  completed 
by  the  team  to  provide  CMMI®-DEV  vl.3  equivalent  ratings. 

•  Ratings  against  CMMI®-  DEV  vl.3  helped  identify  areas  where 
the  potential  changes  in  the  model  would  impact  the 
organization  being  appraised. 
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Candidate  Findings  for 
CMMI-DEV  vl.3 


•  The  maintenance  of  traceability  to  business  objectives  for  the 
organization’s  quantitative  objectives  for  quality  and  process 
performance  was  not  explicitly  evident.  [OPP  SP  1.2] 

•  Attributes  critical  to  evaluating  performance  to  help  achieve 
the  project’s  quality  and  process  performance  objectives  were 
not  clearly  identifiable.  [OPP  SP  1.3] 

•  Risks  associated  with  achieving  the  project’s  quality  and 
process  performance  objectives  are  not  explicitly  identified  or 
managed.  [QPM  SP  2.1] 

•  Incremental  improvements  are  not  explicitly  called  out  as 
relating  to  the  QPPOs  and  business  objectives.  [OPM  SP2.1] 
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•  In  evaluating  CMMI-DEV  vl.3  we  had  the  biggest  problem  with 
OPM. 

-  We  were  able  to  agree  that  the  second  two  goals  would  be 
valuable  with  some  rewording. 

•  There  was  a  lot  of  discussion  about  “Review  Business 
Objectives”  in  OPM. 

-  We  suggested  moving  Establish  and  Maintain  Business 
Objectives  to  OPF. 

-  We  suggested  moving  Maintain  Business  Objectives  to  OPP. 

-  We  suggested  removing  Review  Business  Objectives  from  OPM 
because  it  is  already  in  OPP. 
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Lessons  Learned 


•  Planning  for  and  obtaining  the  necessary  resources  to  train 
the  team,  while  the  model  was  still  being  developed,  was  not 
easy. 

•  Using  draft  material  from  a  model  with  a  moving  baseline  was 
difficult  to  manage  and  caused  rework  for  the  team. 

•  A  template  or  any  specific  guidance  on  what  is  the  desired 
output  for  piloting  a  new  model  would  have  been  helpful. 
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•  We  completed  a  thorough  review  of  CMMI®-  DEV  vl  .2  and 
CMMI®-  DEV  vl.3  through  the  eyes  of  an  appraisal  team  and 
the  implementing  organization. 

•  The  team  agreed  with  most  of  the  changes  in  CMMI®-  DEV  vl  .3 
and  felt  that  the  changes  clarified  the  intent  of  the  practices. 

•  Our  recommendations  were  provided  directly  to  the  CMMI® 
development  team  immediately  upon  completing  the  appraisal 
to  support  their  decisions  for  further  development  of  the  final 
draft. 

•  Most  of  our  pilot  recommendations  were  considered  and 
incorporated  into  the  model. 
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